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CONTROLLED STOCHASTIC NETWORKS IN HEAVY TRAFFIC: 
CONVERGENCE OF VALUE FUNCTIONS 

By Amarjit Budhiraja 1 and Arka P. Ghosh 2 

University of North Carolina and Iowa State University 

Scheduling control problems for a family of unitary networks un- 
der heavy traffic with general interarrival and service times, proba- 
bilistic routing and an infinite horizon discounted linear holding cost 
are studied. Diffusion control problems, that have been proposed as 
approximate models for the study of these critically loaded controlled 
stochastic networks, can be regarded as formal scaling limits of such 
stochastic systems. However, to date, a rigorous limit theory that jus- 
tifies the use of such approximations for a general family of controlled 
networks has been lacking. It is shown that, under broad conditions, 
the value function of the suitably scaled network control problem con- 
verges to that of the associated diffusion control problem. This scaling 
limit result, in addition to giving a precise mathematical basis for the 
above approximation approach, suggests a general strategy for con- 
structing near optimal controls for the physical stochastic networks 
by solving the associated diffusion control problem. 

1. Introduction. As an approximation to control problems for critically- 
loaded stochastic networks, Harrison (in [16], see also [18, 20]) has formu- 
lated a stochastic control problem in which the state process is driven by 
a multidimensional Brownian motion along with an additive control that 
satisfies certain feasibility and nonnegativity constraints. This control prob- 
lem, that is, usually referred to as the Brownian Control Problem (BCP) 
has been one of the key developments in the heavy traffic theory of con- 
trolled stochastic processing networks (SPN). BCPs can be regarded as for- 
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mal scaling limits for a broad range of scheduling and sequencing control 
problems for multiclass queuing networks. Finding optimal (or even near- 
optimal) control policies for such networks — which may have quite general 
non-Markovian primitives, multiple server capabilities and rather complex 
routing geometry — is in general prohibitive. In that regard, BCPs that pro- 
vide significantly more tractable approximate models are very useful. In this 
diffusion approximation approach to policy synthesis, one first finds an opti- 
mal (or near-optimal) control for the BCP which is then suitably interpreted 
to construct a scheduling policy for the underlying physical network. In re- 
cent years there have been many works [1, 3, 4, 8, 11, 12, 26, 28] that consider 
specific network models for which the associated BCP is explicitly solvable 
(i.e., an optimal control process can be written as a known function of the 
driving Brownian motions) and, by suitably adapting the solution to the un- 
derlying network, construct control policies that are asymptotically (in the 
heavy traffic limit) optimal. The paper [25] also carries out a similar pro- 
gram for the crisscross network where the state-space is three dimensional, 
although an explicit solution for the BCP here is not available. 

Although now there are several papers which establish a rigorous connec- 
tion between a network control problem and its associated BCP by exploiting 
the explicit form of the solution of the latter, a systematic theory which jus- 
tifies the use of BCPs as approximate models has been missing. In a recent 
work [9] it was shown that for a large family of Unitary Networks (follow- 
ing terminology of [7], these are networks with a structure as described in 
Section 2), with general interarrival and service times, probabilistic routing 
and an infinite horizon discounted linear holding cost, the cost associated 
with any admissible control policy for the network is asymptotically, in the 
heavy traffic limit, bounded below by the value function of the BCP. This 
inequality, which provides a useful bound on the best achievable asymptotic 
performance for an admissible control policy, was a key step in developing 
a rigorous general theory relating BCPs with SPN in heavy traffic. 

The current paper is devoted to the proof of the reverse inequality. The 
network model is required to satisfy assumptions made in [9] (these are sum- 
marized above Theorem 2.10). In addition, we impose a nondegeneracy con- 
dition (Assumption 2.12), a condition on the underlying renewal processes 
regarding probabilities of deviations from the mean (Assumption 2.13) and 
regularity of a certain Skorohod map (Assumption 2.15) (see next para- 
graph for a discussion of these conditions). Under these assumptions we 
prove that the value function of the BCP is bounded below by the heavy 
traffic limit (limsup) of the value functions of the network control problem 
(Theorem 2.16). Combining this with the result obtained in [9] (see The- 
orem 2.10), we obtain the main result of the paper (Theorem 2.18). This 
theorem says that, under broad conditions, the value function of the net- 
work control problem converges to that of the BCP. This result provides, 
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under general conditions, a rigorous basis for regarding BCPs as approxi- 
mate models for critically loaded stochastic networks. 

Conditions imposed in this paper allow for a wide range of SPN models. 
Some such models, whose description is taken from [7], are discussed in detail 
in Examples l(a)-(c). We note that our approach does not require the BCP 
to be explicitly solvable and the result covers many settings where explicit 
solutions are unavailable. Most of the conditions that we impose are quite 
standard and we only comment here on three of them: Assumptions 2.5, 
2.6 and 2.15. Assumption 2.5 says that each buffer is processed by at least 
one basic activity (see Remark 2.4). This condition, which was introduced 
in [7], is fundamental for our analysis. In fact, [7] has shown that without 
this assumption even the existence of a nonnegative workload matrix may 
fail. Assumption 2.6 is a natural condition on the geometry of the underlying 
network. Roughly speaking, it says that a nonzero control action leads to 
a nonzero state displacement. Assumption 2.15 is the third key requirement 
in this work. It says that the Skorohod problem associated with a certain 
reflection matrix D [see equation (2.43) for the definition of D] is well posed 
and the associated Skorohod map is Lipschitz continuous. As Example 1 
discusses, this condition holds for a broad family of networks (including all 
multiclass open queuing networks, as well as a large family of parallel server 
networks and job-shop networks). 

The papers [1, 3, 4, 8, 12, 28] noted earlier, that treat the setting of ex- 
plicitly solvable BCP, do much more than establish convergence of value 
functions. In particular, these works give an explicit implementable con- 
trol policy for the underlying network that is asymptotically optimal in the 
heavy traffic limit. In the generality treated in the current work, giving ex- 
plicit recipes (e.g., threshold type policies) is unfeasible, however, the policy 
sequence constructed in Section 4.1 suggests a general approach for building 
near asymptotically optimal policies for the network given a near optimal 
control for the BCP. Obtaining near optimal controls for the BCP in general 
requires numerical approaches (see, e.g., [23, 24, 27]), discussion of which is 
beyond the scope of the current work. 

We now briefly describe some of the ideas in the proof of the main result — 
Theorem 2.16. We begin by choosing, for an arbitrary e > 0, a suitable e- 
optimal control Y for the BCP and then, using Y, construct a sequence 
of control policies {T r } r >i for the network model such that the (suitably 
scaled) cost associated with T r converges to that associated with Y, as 
r — > oo. This yields the desired reverse inequality. One of the key difficulties 
is in the translation of a given control for the BCP to that for the physical 
network. Indeed, a (near) optimal control for the BCP can be a very gen- 
eral adapted process with RCLL paths. Without additional information on 
such a stochastic process, it is not at all clear how one adapts and applies 
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it to a given network model. A control policy for the network needs to spec- 
ify how each server distributes its effort among various job classes at any 
given time instant. By a series of approximations we show that one can find 
a rather simple e-optimal control for the BCP, that is, easy to interpret and 
implement on a network control problem. As a first step, using PDE charac- 
terization results for general singular control problems with state constraints 
from [2] (these, in particular, make use of the nondegeneracy assumption — 
Assumption 2.12), one can argue that a near-optimal control can be taken to 
be adapted to the driving Brownian motion and be further assumed to have 
moments that are sub exponential in the time variable (see Lemma 3.10). 
Using results from [10], one can perturb this control so that it has continu- 
ous sample paths without significantly affecting the cost. Next, using ideas 
developed by Kushner and Martins [25] in the context of a two-dimensional 
BCP, one can further approximate such a control by a process with a fixed 
(nonrandom) finite number of jumps that take values in a finite set. Two 
main requirements (in addition to the usual adaptedness condition) for such 
a process to be an admissible control of a BCP (see Definition 2.9) are the 
nonnegativity constraints (2.39) and state constraints (2.38). It is relatively 
easy to construct a pure jump process that satisfies the first requirement 
of admissibility, namely, the nonnegativity constraints, however, the nonde- 
generate Brownian motion in the dynamics rules out the satisfaction of the 
second requirement, that is, state constraints, without additional modifica- 
tions. This is where the regularity assumption on a certain Skorohod map 
(Assumption 2.15) is used. The pure jump control is modified in a manner 
such that in between successive jumps one uses the Skorohod map to employ 
minimal control needed in order to respect state constraints. Regularity of 
the Skorohod problem ensures that this modification does not change the 
associated cost much. The Skorohod map also plays a key role in the weak 
convergence arguments used to prove convergence of costs. The above con- 
struction is the essential content of Theorem 3.5. The e-optimal control that 
we use for the construction of the policy sequence requires two additional 
modifications [see part (iii) of Theorem 3.5 and below (3.14)] which facili- 
tate adapting such a control for the underlying physical network and in some 
weak convergence proofs, but we leave that discussion for later in Section 3 
(see Remark 3.6 and above Theorem 3.8). 

Using a near-optimal control Y of the form given in Section 3 (cf. Theo- 
rem 3.8), we then proceed to construct a sequence of policies {T r } for the 
underlying network. The key relation that enables translation of Y into {T r } 
is (2.16) using which one can loosely interpret Y(t) as the asymptotic de- 
viation, with suitable scaling, of T r {t) from the nominal allocation x*t (see 
Definition 2.2 for the definition of nominal allocation vector). Recall that Y 
is constructed by modifying, through a Skorohod constraining mechanism, 
a pure jump process (say, Yq). In particular, Y has sample paths that are, 
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in general, discontinuous. On the other hand, note that an admissible pol- 
icy T r is required to be a Lipschitz function (see Remark 2.8). This suggests 
the following construction for T r . Over time periods (say, At) of constancy 
of Yq one should use the nominal allocation (i.e., a;* At), while jump-instants 
should be stretched into periods of length of order r (note that in the scaled 
network, time is accelerated by a factor of r 2 and so such periods translate 
to intervals of length 1/r in the scaled evolution and thus are negligible) 
over which a nontrivial control action is employed as dictated by the jump 
vector (see Figure 4 for a more complete description). This is analogous to 
the idea of a discrete review policy proposed by Harrison [17] (see also [1] 
and references therein). There are some obvious difficulties with the above 
prescription, for example, a nominal allocation corresponds to the average 
behavior of the system and for a given realization is feasible only when the 
buffers are nonempty. Thus, one needs to modify the above construction 
to incorporate idleness, that is, caused due to empty buffers. The effect of 
such a modification is, of course, very similar to that of a Skorohod con- 
straining mechanism and it is tempting to hope that the deviation process 
corresponding to this modified policy converges to Y (in an appropriate 
sense), as r — > oo. However, without further modifications, it is not obvious 
that the reflection terms that are produced from the idling periods under 
this policy are asymptotically consistent with those obtained from the Sko- 
rohod constraining mechanism applied to (the state process corresponding 
to) Yq. The additional modification [see (4.8)] that we make roughly says 
that jobs are processed from a given buffer over a small interval A, only 
if at the beginning of this interval there are a "sufficient" number of jobs 
in the buffer. This idea of safety stocks is not new and has been used in 
previous works (see, e.g., [1, 3, 4, 8, 27]). The modification, of course, in- 
troduces a somewhat nonintuitive idleness even when there are jobs that 
require processing. However, the analysis of Section 4 shows that this idle- 
ness does not significantly affect the asymptotic cost. The above very rough 
sketch of construction of T r is made precise in Section 4.1. 

The rest of the paper is devoted to showing that the cost associated 
with T r converges to that associated with Y. It is unreasonable to expect 
convergence of controls (e.g., with the usual Skorohod topology) — in partic- 
ular, note that T r has Lipschitz paths for every r while Y is a (modification 
of) a pure jump process - however, one finds that the convergence of costs 
holds. This convergence proof, and the related weak convergence analysis, 
is carried out in Sections 4.2 and 4.3. 

The paper is organized as follows. Section 2 describes the network struc- 
ture, all the associated stochastic processes and the heavy-traffic assump- 
tions as well as the other assumptions of the paper. The section also presents 
the SPN control problem, that is, considered here, along with the main re- 
sult of the paper (Theorem 2.18). Section 3 constructs (see Theorem 3.8) 
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a near-optimal control policy for the BCP which can be suitably adapted to 
the network control problem. In Section 4 the near-optimal control policy 
from Section 3 is used to obtain a sequence of admissible control policies for 
the scaled SPN. The main result of the section is Theorem 4.5, which es- 
tablishes weak convergence of various scaled processes. Convergence of costs 
(i.e., Theorem 2.17) is an immediate consequence of this weak convergence 
result. Theorem 2.18 then follows on combining Theorem 2.17 with results 
of [9] (stated as Theorem 2.10 in the current work). Finally, the Appendix 
collects proofs of some auxiliary results. 

The following notation will be used. The space of reals (nonnegative reals), 
positive (nonnegative) integers will be denoted by R (R+), N (No), respec- 
tively. For m > 1 and G (0, oo), C m [Cg n ] will denote the space of continuous 
functions from [0, oo) (resp. [0,0]) to R m with the topology of uniform con- 
vergence on compacts (resp. uniform convergence). Also, T) rn \V^ 1 ] will denote 
the space of right continuous functions with left limits, from [0, oo) (resp. 
[0,0]) to R m with the usual Skorohod topology. For y G V m and t,S > 0, 
we write sup < s < t \y(s)\ = {y^t and sup < Sl < S2 < t; | Sl _ S2 |< 5 |y(si) - y(s 2 )\ = 
Wy{5), where for z = {z\, . . . , z m )' G R m , \z\ 2 = YliLi \ z i\ 2 - AH vector inequal- 
ities are to be interpreted component- wise. We will call a function / G T> m 
nonnegative if f(t) > for all t G R+. A function / G T> m is called nonde- 
creasing if it is nondecreasing in each component. All (stochastic) processes 
in this work will have sample paths that are right continuous and have 
left limits, and thus can be regarded as 2? m -valued random variables with 
a suitable m. For a Polish space £ , B(£) will denote the corresponding Borel 
sigma-field. Weak convergence of (£,£>(£)) valued random variables Z n to Z 
will be denoted as Z n => Z. Sequence of processes {Z n } is tight if and only if 
the measures induced by Z n 's on (V m ,B{V m )) form a tight sequence. A se- 
quence of processes with paths in T> m (m > 1) is called C-tight if it is tight 
in T> m and any weak limit point of the sequence has paths in C m almost 
surely (a.s.). For processes {Z n }, Z defined on a common probability space, 
we say that Z n converge to Z, uniformly on compact time intervals (u.o.c), 
in probability (a.s.) if for all t > 0, sup 0<s<t \Z n (s) — Z(s)\ converges to zero 
in probability (resp. a.s.). To ease the notational burden, standard notation 
(that follow [6, 7]) for different processes are used (e.g., Q for queue-length, / 
for idle time, W for workload process etc.). We also use standard notation, 
for example, W, W, to denote fluid scaled, respectively, diffusion scaled, ver- 
sions of various processes of interest [see (2.21) and (2.22)]. All vectors will 
be column vectors. An m-dimensional vector with all entries 1 will be de- 
noted by l m . For a vector a, diag(a) will denote the diagonal matrix such 
that the vector of its diagonal entries is a. M' will denote the transpose 
of a matrix M. Also, Cj, i = 0,1,2, ... , will denote generic constants whose 
values may change from one proof to the next. 
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2. Multiclass queueing networks and the control problem. Let (Q, J 7 , IP) 

be a probability space. All the random variables associated with the network 
model described below are assumed to be defined on this probability space. 
The expectation operation under P will be denoted by E. 

Network structure. We begin by introducing the family of stochastic pro- 
cessing network models that will be considered in this work. We closely follow 
the terminology and notation used in [4, 6, 7, 16, 18, 20]. The network has I 
infinite capacity buffers (to store I many different classes of jobs) and K 
nonidentical servers for processing jobs. Arrivals of jobs, given in terms of 
suitable renewal processes, can be from outside the system and/or from the 
internal rerouting of jobs that have already been processed by some server. 
Several different servers may process jobs from a particular buffer. Service 
from a given buffer i by a given server k is called an activity. Once a job 
starts being processed by an activity, it must complete its service with that 
activity, even if its service is interrupted for some time (e.g., for preemp- 
tion by a job from another buffer). When service of a partially completed 
job is resumed, it is resumed from the point of preemption — that is, the 
job needs only the remaining service time from the server to get completed 
(preemptive-resume policy). Also, an activity must complete service of any 
job that it started before starting another job from the same buffer. An ac- 
tivity always selects the oldest job in the buffer that has not yet been served, 
when starting a new service [i.e., First In First Out (FIFO) within class]. 
There are J activities [at most one activity for a server-buffer pair (i,k), so 
that J < I • K]. Here the integers I, J,K are strictly positive. Figure 1 gives 
a schematic for such a model. 

Let I = {1, . . . ,1}, J = {1, . . . , J} and K = {1, . . . , K}. The correspondence 
between the activities and buffers, and activities and servers are described 
by two matrices C and A respectively. C is an I x J matrix with CV,- = 1 if the 
jth activity processes jobs from buffer i, and = otherwise. The matrix A 
is K x J with A^j = 1 if the kih. server is associated with the jth activity, 
and Akj = otherwise. Each activity associates one buffer and one server, 
and so each column of C has exactly one 1 (and similarly, every column 
of A has exactly one 1). We will further assume that each row of C (and ^4) 
has at least one 1, that is, each buffer is processed by (server is processing, 
resp.) at least one activity. For j G J, let a(j) = (o"i(j), 02 (i)) = (hk), if 
activity j corresponds to the kth server processing class i jobs. Let, for 
feel, 3(k) = {j G J : 02 (j) = k} and I(k) = {oi{j):j G J(k)}. Thus, for the 
feth server, S(k) denotes the set of activities that the server can perform, 
and I(k) represents the corresponding buffers from which the jobs can be 
processed. 
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Fig. 1. A network with I buffers, J activities, K servers and probabilistic routing (given 
by the matrix P). 

Stochastic primitives. We are interested in the study of networks that 
are nearly critically loaded. Mathematically, this is modeled by considering 
a sequence of networks {M r } that "approach heavy traffic," as r — > oo, in 
the sense of Definition 2.2 below. Each network in the sequence has identical 
structure, except for the rate parameters that may depend on r. Here r € 
§ C M+, where § is a countable set: {r±,r2, ■ ■ •} with 1 < r\ < r% < • ■ • and 
r n — > oo, as n — > oo. One thinks of the physical network of interest as the 
rth network embedded in this sequence, for a fixed large value of r. For 
notational simplicity, throughout the paper, we will write the limit along 
the sequence r n as n — > oo simply as "r — > oo." Also, r will always be taken 
to be an element of S and, thus, hereafter the qualifier r 6 S will not be 
stated explicitly. 

The rth network N r is described as follows. If the ith class (i £ I) has 
exogenous job arrivals, the interarrival times of such jobs are given by a se- 
quence of nonnegative random variables {ii[(ra):n > 1} that are i.i.d with 
mean and standard deviation l/a[,o""' r € (0,oo) respectively. Let, by re- 
labeling if needed, the buffers with exogenous arrivals correspond to i 6 
{1, . . . ,1'} := F, where I' < I. We set a[,cr"' r = and u[(n) = oo,n > 1, for 
i £ I \ I'. Service times for the jth type of activity (for j £ J) are given by 
a sequence of nonnegative random variables {Vj(n):n > 1} that are i.i.d. 
with mean and standard deviation l//3J,<7j' r S (0,oo) respectively. We will 
assume that the above random variables are in fact strictly positive, that is, 



(2.1) for all i € I, j G J, P«(l) > 0) = > 0) = 1. 
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We will further impose the following uniform integrability condition: 

(2 2) the collection {(<(1)) 2 , (v r j (l)) 2 ; r > l,j £ J,i £ I'} 

is uniformly integrable. 

Rerouting of jobs completed by the jth activity is specified by a sequence 
of (I + 1) -dimensional vector {(<j)Q T (n), (^^(n))' , n > 1}, where (/> J,r (n) = 
(4>l' r (n) :i £ I). For each j £ J and i £ I U {0}, (fP^ (n) = 1 if the nth com- 
pleted job by activity j gets rerouted to buffer i, and takes the value zero 
otherwise, where i = represents jobs leaving the system. It is assumed that 
for each fixed r, {(</>Q ,r (n), (f>>' r (n)), n > 1}, j £ J, are (mutually) independent 
sequences of i.i.d Multinomial(i +1 )(l, (Pq,^)), where p 3 = (pj : i = 1, . . . , I). 

That, in particular, means, for j £ J,n > 1, X^=o ^i' r ( n ) = Si=oPi = 1- -^ ur " 
thermore, for fixed j £ J, ii,«2 G I; 

(2.3) Cov(<(n),^(n)) =aft 2 = -p^ + p{<^ 2 , 

where (5^ i 2 is 1 if %\ = ii and otherwise. We also assume that, for each r, 
the random variables 

(2.4) {<W>^(™)><fto r "H)^' r '( n )> n > 1>* 6 I,j £ J} 
are mutually independent. 

Next we introduce the primitive renewal processes, (E r , S r ), that describe 
the state dynamics. The process (El, . . . , Ey) is the I'-dimensional exoge- 
nous arrival process, that is, for each % £ I', E^(t) is a renewal process which 
denotes the number of jobs that have arrived to buffer i from outside the 
system over the interval [0,t]. For class i to which there are no exogenous 
arrivals (i.e., i £ I\F), we set E?(t) = for all t > 0. We will denote the pro- 
cess (El, . . . ,E^)' by E r . For each activity j £ J, Sj(t) denotes the number 
of complete jobs that could be processed by activity j in [0, t] if the associ- 
ated server worked continuously and exclusively on jobs from the associated 
buffer in [0, t] and the buffer had an infinite reservoir of jobs. The vector 
(SI, . . . , Sj)' is denoted by S r . More precisely, for i £ I, j £ J, m > 1, let 

m m 

(2.5) eiM =E<(™)> =!>;(»)■ 

n=l n=l 

We set £[(0) = 0,r/J(0) = 0. Then E\, Sj are renewal processes given as 
follows. For t > 0, 

(2.6) E^(t) = max{m>0:Ci(m)<t}, = max{m > 1 :r%(m) < t}. 

Finally, we introduce the routing sequences. Let <J?^' r (n) denote the number 
of jobs that are routed to the ith buffer, among the first n jobs completed 
by activity j. Thus, for i £ I, j £ J, 

n 

(2.7) *f (n) = 5>f(m), n = l,2,.... 

m=l 
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We will denote the I-dimensional sequence {(<&{' r (n), . . . , <&j' r (n))'} corre- 
sponding to routing of jobs completed by the jth activity by {<& J ' r (n)}. 
Also, * r (n) will denote the I x J matrix ($> l ' r (n), * 2,r (n), . . . , <& J ' r (n)). 

Control. A Scheduling policy or control for the rth SPN is specified by 
a nonnegative, nondecreasing J-dimensional process T r = {(T[(t), . . . , Tj (t))' , 
t > 0}. For any j G J, t > 0, 1J(t) represents the cumulative amount of time 
spent on the jth activity up to time t. For a control T r to be admissible, it 
must satisfy additional properties which are specified below in Definition 2.7. 

State processes. For a given scheduling policy T r , the state processes of 
the network are the associated I-dimensional queue length process Q r and 
the K-dimensional idle time process I r . For each t > 0, i G I, Ql(t) represents 
the queue-length at the ith buffer at time t (including the jobs that are in 
service at that time), and for k = 1, ...,K, IL(t) is the total amount of 
time the Arth server has idled up to time t. Let q r = Q r (0) G N 1 be the I- 
dimensional vector of queue-lengths at time 0. Note that, for j G S,t > 0, 
Sj(TJ(t)) is the total number of services completed by the jth activity up 
to time t. The total number of completed jobs (by activity j) up to time t 
that get rerouted to buffer i equals ^' r (5J(TJ(t))). Recalling the definition 
of matrices C and A, the state of the system at time t > can be described 
by the following equations: 

J J 

(2.8) Q\{t) =q r + El{t) - CijS^Tjit)) + £ *f (SJ(IJ(i))), • G I, 

j=i i=i 

j 

(2.9) I r k (t)=t-J2 A k j TJ(t), keK. 

i=i 

Heavy traffic. We now describe the main heavy traffic assumption[18, 
20] . We begin with a condition on the convergence of various parameters in 
the sequence of networks {AA r }. 

Assumption 2.1. There are q, a, a u G (3, a v G R^., 6\ G K 1 , 02 G M J 
such that /3 > 0, a v > 0, cti, af = if and only if i G I \ I', and, as r — > oo, 

q = >q. 

r 

The definition of heavy traffic, for the sequence {Af r }, as introduced in [20] 
(also see [6, 7, 18]), is as follows. 



(2.10) 



1\ = r(a - a) — > 0\, 



a 
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Definition 2.2 [Heavy traffic]. Define I x J matrices P',R, such that 
P(j =p-, for i € I,j e JJ, and 

(2.11) R = (C-P') diag(^). 

We say that the sequence {M r } approaches heavy traffic as r — > oo if, in 
addition to Assumption 2.1, the following two conditions hold: 

(i) There is a unique optimal solution (x* , p* ) to the following linear 
program (LP): 

(2.12) minimize p such that Rx = a and Ax < /?1k for all x > 0. 

(ii) The pair (x*,p*) satisfies 

(2.13) p* = 1 and Ax* = 1 K . 

Assumption 2.3. The sequence of networks {M r } approaches heavy 
traffic as r — > oo . 

Remark 2.4. From Assumption 2.3, x* given in (i) of Definition 2.2 is 
the unique J-dimensional nonnegative vector satisfying 

(2.14) Rx* = a, Ax* = l K . 

Following [20] , assume without loss of generality (by relabeling activities, if 
necessary), that the first B components of x* are strictly positive (corre- 
sponding activities are referred to as basic) and the rest are zero (nonbasic 
activities). For later use, we partition the following matrices and vectors in 
terms of basic and nonbasic components: 



(2.15) 







r 



± n 



A=[B:N], R=[H:M], 



where T r is some control policy, is a (J — B)-dimensional vector of ze- 
ros, B,N,H,M are K x B, K x (J - B), I x B and I x (J - B) matrices, 
respectively. 

The following assumption (see [7]) says that for each buffer there is an 
associated basic activity. 

Assumption 2.5. For every i 6 I, there is a j £ J such that Rij > and 
x* > 0. 

Other processes. Components of the vector x* defined above can be in- 
terpreted as the nominal allocation rates for the J activities. Given a control 
policy T r , define the deviation process Y r as the difference between T r and 
the nominal allocation: 

(2.16) Y r (t) = x*t-T r (t), t>0. 



12 
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It follows from (2.9) and (2.14) that the idle-time process I r has the following 
representation: 

I r (t)=AY r (t), t>0. 

Let N = K + J — B. Next we define a N x J matrix K and N-dimensional 
process U r as follows: 

B N 



(2.17) K 







U r (t) = KY r (t), t>0, 



(2.18) U r (t) 



t > 0. 



where I denotes a (J — B) x (J — B) identity matrix. Note that, with as 
in (2.15), 

T r n (t) 

Finally, we introduce the workload process W r which is defined as a certain 
linear transformation of the queue-length process and is of dimension no 
greater than of the latter. More precisely, W r is an L-dimensional process 
(L = I + K - B, see [7]) defined as 

(2.19) W r (t) = AQ r (t), t>0, 

where A is a L x I-dimensional matrix with rank L and nonnegative entries, 
called the workload matrix. We will not give a complete description of A 
since that requires additional notation; and we refer the reader to [7, 18] for 
details. The key fact that will be used in our analysis is that there is a L x N 
matrix G with nonnegative entries (see (3.11) and (3.12) in [18]) such that 

(2.20) AR = GK. 

We will impose the following additional assumption on G which says that 
each of its columns has at least one strictly positive entry. The assumption 
is needed in the proof of Lemma 3.10 [see (3.36)]. 

Assumption 2.6. There exists a c > such that for every u G R+ , 
\Gu\ > c\u\. 

Rescaled processes. We now introduce two types of scalings. The first 
is the so-called fluid scaling, corresponding to a law of large numbers, and 
the second is the standard diffusion scaling, corresponding to a central limit 
theorem. 

Fluid Scaled Process: This is obtained from the original process by accel- 
erating time by a factor of r 2 and scaling down space by the same factor. 
The following fluid scaled processes will play a role in our analysis. For t > 0, 

E r {t) = r- 2 E r (r 2 t), S r (t) = r~ 2 S r \r 2 t), 

(2.21) * r (t)=r^ 2 * r (Lr 2 tJ), f r {t) = r~ 2 T r (r 2 t), 
F(t) = r- 2 T r (r 2 t), Q r (t) = r~ 2 Q r (r 2 *). 
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Here for x G M + , |_^J denotes its integer part, that is, the greatest integer 
bounded by x. 

Diffusion Scaled Process: This is obtained from the original process by 
accelerating time by a factor of r 2 and, after appropriate centering, scaling 
down space by r. Some diffusion scaled processes that will be used are as 
follows. For t > 0, 

^ (gV^-og) [ § {S r {rH)-^r 2 t) ^ 
r r 



¥{t) ^(^([rH\])-[r 2 t\P') 



(2.22) 



r 

U r (t) = r~ l U r (r 2 t), Q r (t) = r^Q^rH), 
W r (t) = r- 1 W r (r 2 t), Y r {t) = r^Y^rH) 



The processes U r ,Q r ,W r are not centered, as one finds (see Lemma 3.3 
of [9]) that, with any reasonable control policy, their fluid scaled versions 
converge to zero as r — > oo. Define for t > 0, 

J 

XT(t) = El(t) - Y,(Cij - p?')Sr(2J(*)) 

(2.23) 

-^*f(5J(Tj(t))), i€L 
j'=i 

Recall (9[ and g r from Assumption 2.1. Using (2.8), (2.9), (2.14) and (2.17), 
one has the following relationships between the various scaled quantities 
defined above. For all t > 0, 

Q r (t) = C(t)+RY r (t), U r (t) = KY r (t), 

where 

(2.24) C'(t) =q r + X r (t) + [0Jt - (C - P') diag(^)^(t)]. 
Also, using (2.19), (2.20) and (2.24), for all t > 0, 

(2.25) W r {t) = Aq r + AA r (i) + A[0[t - (C - P') diag(^)T r (i)] + GU r {t). 

Admissibility of control policies. The definition of admissible policies 
(Definition 2.7), given below, incorporates appropriate nonanticipativity re- 
quirements and ensures feasibility by requiring that the associated queue- 
length and idle-time processes (Q r ,P) are nonnegative. 

For m = (mi, . . . , mi) G N 1 , n = (m, . . . , nj) G N J we define the multipa- 
rameter filtration generated by interarrival and service times and routing 
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variables as 

F((m,n)) 

(2.26) 

= <r{uU m 'i)i v j( n j)M' r ( n j) :m i ^ m i' n 'j <nj-,i£ l,j G J}. 

Then {F r {{m,n)) :m G N\?i G N 3 } is a multiparameter nitration with the 
following (partial) ordering: 

(m , n 1 ) < (m 2 , n 2 ) if and only if < m 2 ,nj < n 2 ;z G I, j G J. 

We refer the reader to Section 2.8 of [15] for basic definitions and properties 
of multiparameter nitrations, stopping times and martingales. Let 

(2.27) r= \/ F{{m,n)). 

For all (m,n) G {0, 1} I+J , we define J- r ((m,n)) = T r ((l, 1)) where 1 denotes 
the vector of l's. It will be convenient to allow for extra randomness, than 
that captured by T r , in formulating the class of admissible policies. Let Q be 
a ex-field independent of J- r . For m G N 1 , n G N J , let J- r ((m, n)) = Tg(ra, n) = 
P\{m,n))vg. 

Definition 2.7. For a fixed r and q r G R?j_, a scheduling policy T r = 
{ (T-f (i) , . . . , Tj (i) ) : t > 0} is called admissible for M r with initial condition q r 
if for some Q independent of T r ', the following conditions hold: 

(i) TJ is nondecreasing, nonnegative and satisfies TJ(0) = for j G J. 

(ii) l r k defined by (2.9) is nondecreasing, nonnegative and satisfies 11.(0) = 
for k = l,...,K. 

(iii) Q\ defined in (2.8) is nonnegative for i G I. 

(iv) Define for each r, t > 0, 

( TS(t) = K^(t),^ S (i)) 

(2.28) 

= (£[(r 2 i) + 1 : * G I; ^(7j(r 2 i)) + 1 : j G J). 
Then, for each t > 0, 

(2.29) <7o(*) is a {-^(("^ n)):m£ N 1 , n G N J } stopping time. 
Define the filtration {^[(t) :t > 0} as 

Jftt)=.F'K(t)) 

(2.30) 

= a{^G T r :AC\{al{t) < (m, n)} G T r ((m, n)),m G N 1 , n G N J }. 

Then 

(2.31) ET is {J r 1 r (t)}-adapted. 
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Denote by A r (q r ) the collection of all admissible policies for N r with initial 
condition q r . 

Remark 2.8. (i) and (ii) in Definition 2.7 imply, in view of (2.9) and 
properties of the matrix A, that 

(2.32) 0<Tj(t)-Tj(s)<t- s, jGjJfor allO<s<t<oo. 

In particular, TJ is a process with Lipschitz continuous paths. Condition (iv) 
in Definition 2.7 can be interpreted as a nonanticipativity condition. Propo- 
sition 2.8 and Theorem 5.4 of [9] give general sufficient conditions under 
which this property holds (see also Proposition 4.1 of the current work). 

Cost function. For the network N r , we consider an expected infinite hori- 
zon discounted (linear) holding cost associated with a scheduling policy T r 
and initial queue length vector q r : 

(2.33) /( f r ,T f )=EN° e~^h-Q r (t)dt\ + E^J™ e^p ■ dU r (t)^. 

Here, 7 £ (0, 00) is the "discount factor" and h, an I-dimensional vector 
with each component hi £ (0, 00), i £ I, is the vector of "holding costs" for 
the I buffers. In the second term, p > is an N-dimensional vector. The 
first K block of U corresponds to the idleness process I, and, thus, the 
second term in the cost, in particular, captures the idleness cost. The last 
J — B components of U correspond to the time spent on nonbasic activities. 
Thus, this formulation of the cost allows, in addition to the idleness cost, 
the user to put a penalty for using nonbasic activities. 

The formulation of the cost function considered in our work goes back to 
the original work of Harrison et al. [16, 20]. 

The scheduling control problem for N r is to find an admissible control 
policy T r that minimizes the cost J r . The value function V r for this control 
problem is defined as 

(2.34) V r {q r ) = inf J r {q r ,T r ), q r E M. 

T r eA r (q r ) 

Brownian control problem. The goal of this work is to characterize the 
limit of value functions V r as r — > 00, as the value function of a suitable 
diffusion control problem. In order to see the form of the diffusion control 
problem, we will like to send r — > 00 in (2.24). Using the functional central 
limit theorem for renewal processes, it is easily seen that, for all reasonable 
control policies (see again Lemma 3.3 of [9]), when q r converges to some 
q £ M 1 ^, C r defined in (2.24) converges weakly to 

(2.35) C,=q + X + 9i, 
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where 

(2.36) = 0i - (C -P')diag(9 2 )x*. 

Here i(s) = s,s > is the identity map and X is a Brownian motion with 
drift and covariance matrix 

J 

(2.37) E = S u + (C - P')Z V diag(x*)(C - P')' 



j'=i 



where S" is a I x I diagonal matrix with diagonal entries (af) ,i G I, 
is a J x J diagonal matrix with diagonal entries (<rj) 2 ,j G J and E^s are 

I x I matrices with entries (rf?-,ii,i2 G I [see (2.3)]. Although the process Y r 
in (2.24), for a general policy sequence {T r }, need not converge, upon for- 
mally taking limit as r — > oo, one is led to the following diffusion control 
problem. 

Definition 2.9 [Brownian Control Problem (BCP)]. A J-dimensional 
adapted process Y , defined on some filtered probability space (Cl, T , P, {^(t)}) 
which supports an I-dimensional {J-"(£)}-Brownian motion X with drift 
and covariance matrix T, given by (2.37), is called an admissible control 
for the Brownian control problem with the initial condition q G iff the 
following two properties hold P-a.s.: 

(2.38) Q{t) = C(i) + HY(t) > where ((t) = q + X(t) + 8t,t > 0, 

(2.39) U = KY is nondecreasing and U (0) > 0, 

where Q and are as in (2.35) and (2.36) respectively. We refer to $ = (O, J 7 , 
P, {F{t)}, X) as a system. We denote the class of all such admissible controls 
by A(q). The Brownian control problem is to 



/ e-^h-Q(t)dt+ / e-^p-dU{t) 

JO J\0,oo) 



(2.40) infimize J(q,Y) =E 

'[0,oo) 

over all admissible controls Y G A(q). Define the value function 

(2.41) J*(q)= inf J(q,Y). 

YdA(q) 

Recall our standing assumptions (2.1), (2.2), (2.4), Assumptions 2.1, 2.3, 
2.5 and 2.6. The following is the main result of [9]. 

Theorem 2.10 (Budhiraja and Ghosh [9], Theorem 3.1, Corollary 3.2). 
Fix q G and for r > 0, ^GN 1 such that q r — > q as r — )• oo. Then 

\immiV r {q r ) > J*(q). 
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Remark 2.11. The proof in [9] is presented for the case where in the def- 
inition of V r (q r ) [see (2.34)], A r {q r ) is replaced by the smaller family A r (q r ) 
which consists of all T r E A r (q r ) that satisfy (iv) of Definition 2.7 with 
J- 71 ((m,n)) replaced by ^{(m^n)). Proof for the slightly more general set- 
ting considered in the current paper requires only minor modifications and, 
thus, we omit the details. 

For the main result of this work, we will need additional assumptions. 

Assumption 2.12. The matrix S is positive definite. 

We will make the following assumption on the probabilities of deviations 
from the mean for the underlying renewal processes. Similar conditions have 
been used in previous works on construction of asymptotically optimal con- 
trol policies [1, 3, 4, 8, 12]. 

Assumption 2.13. There exists m > 2 and, for each 8 > 0, some c(<5) E 
(0, oo) such that, for j E JJ, i E I, r > 1, t E (1, oo), 



P(|S?(t)-#t|>5t)< 
F(\El(t)-alt\>6t)< 

n\$f r (sr(t))-pi^t\>5t)< 



£(*) 
t m ' 

£0) 

£(£ 
t m ' 



The third inequality above is a consequence of the first two, but we note 
it explicitly here for future use. The assumption is clearly satisfied when Ei 
and Sj are Poisson processes. For general renewal processes, such inequalities 
hold under suitable moment conditions on the interarrival and service time 
distributions. Indeed, if for some m > 1 



2m 



E[supK(l)] 

(2.42) 

l2m 



< OO, 



E 



sup[^'(l)] 2m <oo for all i El, j€ J, 



then, from Theorem 4 of [21], Assumption 2.13 is satisfied. 

We now introduce an assumption on the regularity properties of a cer- 
tain Skorohod map. This map plays a crucial role in our analysis; see proofs 
of Theorems 3.5 and 4.5 [see in particular, (3.13), (3.39), discussion be- 
low (3.45) and the proof of Theorem 3.8]. Let T>\ = {x E T> 1 :x(0) > 0} and 

(2.43) D = {C- P') diag(/3) diag(x*)C". 



18 A. BUDHIRAJA AND A. P. GHOSH 

Definition 2.14. Given x E V l + , we say (z,y) £ V 1 x V 1 solve the 
Skorohod Problem (SP) for (x,D) if: (i) z(0) = x(0), (ii) z = x + Dy, (hi) y is 
nondecreasing and y(0) > 0, (iv) z(t) > for all t > 0, and 

( v ) /[o,oo) 1 {^' l (t)>0} dyitt) = for a11 * G 1 

Denoting by T>q the set of x G such that there is a unique solution 
to the SP for (x, D), we define maps T : T>q — > 2? 1 , T : 2?o — ^ as r(z) = z, 
T(x) = y if (z,y) solve the SP for (x,D). 

Assumption 2.15. V = T>\ and the maps T,f are Lipschitz, namely, 
there exists L € (0, oo) such that for all x\,X2 £ 

sup {|r(x 1 )(t)-r(x 2 )(t)| + |f(x 1 )(t)-f(x 2 )(t)|}<L sup \ Xl {t)-x 2 {t)\. 

0<t<oc 0<t<oo 

We refer the reader to [13, 14] and [19] for sufficient conditions under which 
the above regularity property of the Skorohod map holds. See also Example 1 
below. For later use we introduce the notation T(x) = diag(a;*)C"r(x) for 
x G Since A has nonnegative entries and x j = for j = B + l,...,J, we 
see from the definition of K [see (2.17)] that 

(2.44) if y = T(x), then Ky > 0. 

For rest of the paper, in addition to the assumptions listed above Theo- 
rem 2.10, Assumptions 2.12, 2.13 and 2.15 will be in force. The main result 
of the paper is the following. 

Theorem 2.16. Fix q G Let for r > 0, q r G be such that q r — ^ q 
as r — > oo . Then 

limsupV r (g r )< J*(q). 

r— >oo 

The theorem is an immediate consequence of Theorem 2.17 below which 
is proved in Section 4. For e > 0, we say Y G A(q) is e-optimal for the BCP 
with initial value q if 

J(q,Y)<J*(q) + e. 

When clear from the context, we will omit the phrase "for the BCP with 
initial value g" and merely say that Y is e-optimal. 

Theorem 2.17. Fix q G M+. For r > 0, let q r G be such that q r — > q 
as r — > oo. For every e > ; there exists Y G A(q), which is e-optimal, and 
a sequence T r G A r (q r ), r > 1 such that 

J r (q r ,T r )^ J(q,Y) as r -> oo. 
Combining Theorems 2.10 and 2.16, the following is immediate. 
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Theorem 2.18. Fix q G R\. For r > 0, let q r G be such that q r -)■ q 
as r — > oo. Then, as r — > oo, V r (g r ) — )■ J*(q). 

Assumptions made in this work can loosely be divided into two categories: 
Assumptions on the underlying stochastic primitives, which include, in par- 
ticular, the heavy traffic conditions (Assumptions 2.1, 2.3, 2.12 and 2.13), 
and assumptions made on the network structure (Assumptions 2.5, 2.6 
and 2.15). Below we discuss the validity of these structural assumptions 
for some basic families of SPN models. 



Example 1. The following examples have been described in detail in [7]. 
We will assume here, without loss of generality, that (3j > for all j G J (an 
activity j for which f3j = can simply be deleted from the network descrip- 
tion). Furthermore, for all three settings considered below, Assumption 2.5 
can be made without loss of generality, since otherwise one can consider 
a reduced system obtained by omitting the buffers that are not processed 
by any basic activity. Assumption 2.6 states that the matrix G can be chosen 
in a manner such that it has no columns that are identically zero. Roughly 
speaking, it says that a nonzero control action leads to a nonzero state dis- 
placement. Although this appears to be a very natural geometric condition 
and is trivially satisfied for networks in part (a) below, it is not clear that it 
holds always for examples in parts (b) and (c) below. We will assume this 
condition to hold without further comment. 

Thus, in discussion below, we will focus only on Assumption 2.15. 

(a) Open multiclass queueing networks: These correspond to a setting 
where each buffer is processed by exactly one activity and, consequently, 
there is a one-to-one correspondence between activities and buffers, that is, 
J = I (see left figure in Figure 2 for an example). For such networks, R is 
an I x I-matrix of the form R = (I — P') diag(/3) where P is a nonnegative 




Fig. 2. Open multiclass network (left) and parallel-server system (right). 
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matrix with spectral radius less than 1. In particular, R is nonsingular, K is 
a K x J matrix with full row rank and one can take A = KRr 1 and G = I. 
Here D = (I — P') diag(/3) diag(x*) and from [19] it is known that for such D 
Assumption 2.15 is satisfied. 

(b) Parallel server networks: For such SPN, a buffer can be served by 
more than one activity, however, each job gets processed exactly once before 
leaving the system (i.e., there is no rerouting). See right figure in Figure 2 
for an example. In particular, P = and, hence, R = Cdiag(/3). In this case, 
D = Cdiag(/3)diag(x*)C / = diag( 7 *), where 7* = Ej-.a^ifytf, for * G 1 
From Assumption 2.5 (which, as was noted above, can be made without 
loss of generality) we have that 7* > and, thus, D is a diagonal matrix 
with strictly positive diagonal entries. Assumption 2.15 is clearly satisfied 
for such matrices. 

(c) Job-shop networks: This subclass of networks combines features of 
both (a) and (b): A buffer can be processed by more than one activity and 
jobs, once served, can get rerouted to another buffer for additional process- 
ing. See Figure 3 for an example. Following specific examples considered 
in [22] (see also [7]), we define job-shop networks as those which satisfy the 
following property: If for some j,j' £ J, and i El, cr\{j) = = i, then 
Pji' = Pj'i' f° r au 6 !• Namely, jobs corresponding to any two activities that 
process the same buffer i have an identical (probabilistic) routing structure, 
following their completion by the respective servers. It is easily checked that 
in this case D = (I — P')diag(7*), where 7* is as introduced in (b) and P 
is an I x I-dimensional matrix with entries pai = pji> where j € J is such 
that o~i(j) = i. Under the condition that P has spectral radius less than 1, 
it follows from [19] that Assumption 2.15 is satisfied. 
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3. Near-optimal controls for BCP. The rest of the paper is devoted to 
the proof of Theorem 2.17. Toward that goal, in this section we construct 
near-optimal controls for the BCP with certain desirable features. This con- 
struction is achieved in Theorem 3.8, which is the main result of this section. 

Since an admissible control is not required to be of bounded variation, the 
BCP is a somewhat nonstandard diffusion control problem and is difficult 
to analyze directly. However, as shown in [20], under assumptions made in 
this paper, one can replace this control problem by an equivalent problem of 
Singular Control with State Constraints (SCSC). This control problem, also 
referred to as the Equivalent Workload Formulation (EWF) of the BCP, is 
given below. We begin by introducing the cost function, that is, optimized 
in this equivalent control problem. 

Effective cost function: Recall the definition of the workload matrix A 
introduced in (2.19). Let W = {Az : z £ M.\}. For each w £ W, define 

(3.1) h(w)=mi{h-q:Aq = w,q>0}. 

Since h > 0, the infimum is attained for all w £ W. It is well known (see 
Theorem 2 of [5]) that one can take a continuous selection of the minimizer 
in the above linear program. That is, there is a continuous map q* : W — >• 
such that 

(3.2) Q*( w ) ^ argmin{/i • q : Aq = w, q > 0}. 

q 

Thus, in particular, h is continuous. One can check that h satisfies linear 
lower and upper bounds. In order to see this, define 

(3.3) q*(w) = <f (w)l {H < 1} + \w\q* (J^) hH^}- 

Then (3.2) holds with q* replaced by q* . Since h(w) = h ■ q*(w) and h > 0, 
we have from the above display that 

(3.4) h\w\-b 2 <\h(w)\<b 3 (l + \w\), weW 

for some b\, 62, 63 £ (0, 00). Also, uniform continuity of q* on {w £ W : \ w\ < 1} 
shows that 

\h(wi) - h(w2)\ < m{5){\ + \w\\ + l^l), 

(3.5) 

W\,W2 £ W, |u>l — W2\ < 8. 

Here m is a modulus, that is, a nondecreasing function from [0, 00) —> [0,oo) 
satisfying m(0+) = 0. Inequalities (3.4) and (3.5) will be used in order to 
appeal to some results from [2, 10] (see Remark 3.4 below). Define 

(3.6) IC = {ueM. N \u = Ky,yeR 3 }. 

The Equivalent Workload Formulation (EWF) and the associated control 
problem are defined as follows. 
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Definition 3.1 [Equivalent Workload Formulation (EWF)]. An N-di- 
mensional adapted process U, denned on some filtered probability space (f2, 
J-,¥,{^F(t)}) which supports an I-dimensional {^(^j-Brownian motion X 
with drift and covariance matrix £ defined in (2.37), is called an admissi- 
ble control for the EWF with initial condition w £ W iff the following two 
properties hold P-a.s.: 

U is nondecreasing, U(0) > 0, U(t) £ tC for all t > 0, 

(3.7) 

W(t) = w + A0t + AX(t) + GU(t) £ W for all t > 0, 

where 9 is as in (2.36). We denote the class of all such admissible controls 
by Aq(w). The control problem for the EWF is to 

(3.8) infimize J (w,U)=E / e'^hiW [t]) dt + t I e _7 *p • dU(t), 

JO 7 [0,oo ) 

over all admissible controls U £ Ao(w). Define the value function 

(3.9) Jq{w) = _ inf J (w,U). 

U&Ao(w) 

From Theorem 2 of [20] it follows that for all w £ W, q £ satisfying 
to = Ag, 

(3.10) J*(<7) = J *H. 

The following lemma will be used in order to appeal to some results from [2, 
10]. The proof is based on arguments in [7]. Let K, + = K,C\ 

Lemma 3.2. The cones KL+ and GK.+ have nonempty interiors and W°n 
GK + ^0. 

Proof. From [7] (see above Corollary 7.4 therein) it follows that H 
has full row rank and so there is a B x I matrix such that HH^ = I. 
Let X}, = H^li and let so £ (0,oo) be sufficiently small such that for all 
e £ (0,e ], %l = x* b +ex b > 0. Let a = a + e li and x = [x e b °,0]' £ R 3 . Then 
Rxq = ao. We will now argue that t9 = Aao £ W° n (G7C+)°. Since the rows 
of A are linearly independent, we can find an I x L matrix such that 
AA" 1 " = I. Fix 5 = 2jfty • Then, whenever # £ K L , \$\ < ft we have a + A^-d £ 

R\ and so d + # = A(a + A+tf) £ W. This shows that <d £ W°. Next note 
that i? = Aao = Ai?xo = GKxq. Since ATxo = Bx £ b ° and > 0, we have that 
Kxq £ K, + and so i? = GKxq £ GK, + . Since > 0, we can find E\ £ (0,oo) 
such that whenever Xj, £ M B is such that \xb\ < e±, x b ° + x b > 0. Now fix 
ft = f | At Then, for any Q £ M L with |0| < ft and 5 = [ijtAttf, 0]' £ R 3 , 

+ = A(a + A f tf) = A(i?x + HH ] A ] d) = A(Rx + Rx) = G(K(x + 5)). 
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Since I^A^I < e\, we have (xq + x)j > for all j = 1, . . . ,B. Also, (xo + 
x)j = for all j = B + 1, . . . , J. Thus, K(x + x) G /C+ and, therefore, •& + ■& G 
GK+. It follows that ■& G {GK + )°. 

Finally, we show that /C° ^ 0. Since Ax* = Ik, every row of B must con- 
tain at least one strictly positive entry. Thus, Bx e b ° > 0. Choose £2 G (0,oo) 
sufficiently small such that Bx £ b ° — £2-/V1j_b > 0. Let x = [x e b ° , — ^Ij-b]' G 
]R J . We now argue that u = Kx G (/C + )°. Note that by construction u > 0. 
Thus, we can find £3 > such that u + u > whenever -u G M N satisfies 
1^1 < £3- Also, since has full row rank (see Corollary 6.2 of [7]), we can 
find a J x N matrix such that KK^ = I. Thus, u + u = K(x + -ftT^w) and, 
consequently, n + n G ZC+. The result follows. □ 

Note that the vector xo constructed in the proof of the lemma above has 
the property that Rxq > and Kxq > 0. Thus, we have shown the following: 

Corollary 3.3. The set T = {y G R j : Ky >0,Ry> 0} is nonempty. 

The above result will be used in the construction of a suitable near optimal 
control policy for the BCP [see below (3.14)]. 

Remark 3.4. We will make use of some results from [2] and [10] that 
concern a general family of singular control problems with state constraints. 
We note below some properties of the model studied in the current paper 
that ensure that the assumptions of [2] and [10] are satisfied: 

(a) G has full row rank. This follows from the observation that K, A 
and R have full row ranks and, therefore, 

rank(G) = rank(GK) = rank(Ai?) = rank(A) = L. 

(b) W° fl (G7C+) 7^ and ZC+ has a nonempty interior (see Lemma 3.2). 

(c) (Gu) ■ 1l > \Gu\, u ■ In > \u\ for all u G /C + and w ■ 1l > \w\ for all 
w G W. This is an immediate consequence of the fact that the entries of G 
and A are nonnegative [see above (2.20)]. 

(d) Since A has full row rank and, by Assumption 2.12, S is positive 
definite, we have that ASA' is positive definite. 

The above properties along with Assumption 2.6, (3.4) and (3.5) ensure that 
Assumptions of [2] and [10] are satisfied in our setting. In particular, As- 
sumption (2.1)-(2.2) and (2.8)-(2.10) of [2] hold in view of properties (b), 
(c) and (d) and equations (3.4) and (3.5). Similarly, Assumptions (1), (5) 
and 2.2 of [10] hold in our setting [from property (c), (3.4) and Assump- 
tion 2.6, resp.]. Henceforth, when appealing to results from [2] and [10], we 
will not make an explicit reference to these conditions. 
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Recall £(t) and the map T introduced above (2.35) and (2.44), respec- 
tively. The following is a key step in the construction of a near-optimal 
control with desirable properties. 

Theorem 3.5. Fix q G R\. For each e € (0,oo), there exists Y^ G A(q), 
given on some system that is e-optimal and has the following properties: 

(3.11) y( 1 )=y (1) + f(c + i?y (1) ), 

where Yq 1 ^ is an adapted process with sample paths in V 3 satisfying the fol- 
lowing: For someT,n,M G (0, oo), po, jo G N, with9 = T/po and k = 6/ jo, 

(i) Y (1) (i) = Y (1 \n9) forte [n9, (n+l)0),n = 0, 1, . . . ,p - 1 andY {1 \t) = 

Y (1 \ P0 9) fort>T = Po 9. 

(ii) Letting S V M = {brj :b G 1? ,\b\r} < M,Kb > 0}, 

dY W(n) = Y^(n9) - y (1) ((n - 1)9) G 

forn = l,...,p and dY {1) {0) = Y (1) (0) = 0. 

(iii) There is an i.i.d sequence of Uniform (over [0, 1]) random variables {U n }, 
that is, independent of and for each n = 1, . . . ,po a measurable map 
w n : M 1 ™-? x [0, 1] —7- S^j, n = 1, . . . ,po, such that the map x i— > w n (x, t) 
is continuous, for a.e. t in [0,1], and 

(3.12) dY (1 \n) = w n (X K (n),U n ), 
where X K {n) = {X(£k) :£ = 1,... ,njo}, n = 0, 1, . . . ,pq. 

Proof of Theorem 3.5 is given in Section 3.1. 

Remark 3.6. The above theorem provides an e-optimal control Y^ l \ 
which is the "constrained" -version of a piecewise constant process y {1) . The 

value of Yq 1 ^ changes only at time-points that are integer multiples of 9 and 
is constant for t > T = p$9. Also, the changes in (the value of) the process 
occur in jumps with sizes that are integer multiples of some r] > and are 
bounded by M. The third property in the theorem plays an important role in 
the weak convergence proof [Theorem 4.5, see, e.g., (4.56)] and says that the 
jump-sizes of this piecewise constant process are determined by the Brown- 
ian motion X sampled at discrete instants {k,2k, . . .} and the independent 
random variable U n ; furthermore, the dependence on X is continuous. The 
continuous dependence is ensured using a mollification argument [see be- 
low (3.46)] that has previously been used in [25]. 

The following lemma is a straightforward consequence of the Lipschitz 
property of the Skorohod map, the linearity of the cost and the state dy- 
namics. Proof is given in the Appendix. 
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Lemma 3.7. There is a c\ G (0,oo) such that, if q G R\,T G (0,oo) 
and y 1 , Y 2 G .A((/) defined on a common filtered probability space are such 
that 

Y\T + ■)- Y*(T) = r(Q i (T) + C(T + •) - C(T)), z = 1,2, 

where Q l is defined by the right-hand side of (2.38) by replacing Y there 
by Y l , then 

\J(q, Y 1 ) - J(q, Y 2 )\ < c^Y 1 - Y 2 ^. 
Define ■& : R\ x R 3 ->■ R 3 as 

<%o, y)=y + f (g + Ryi)(l), go e K^, y g m j . 

Note that i? is a Lipschitz map: that is, for some i?ii p G (0, oo), we have for 

(<7o,y),(go,y)eR^xR J , 

(3.13) |tf(g ,y) - Jf)| < 0np(\qo ~ Qo\ + \v~ v\)- 
Also, since i9(go,0) = 0, we have for (go s J/) G ^+ x R J > 

(3.14) |^(go,y)| <^K P |2/|- 

We now present the near-optimal control that will be used in the proof of 
Theorem 2.17. Recall the set T introduced in Corollary 3.3. Fix e > 0, and 
a unit vector y* G T and define C2 G [l,oo) as 

c 2 = max{2ci(p + 1)(1 + L\ diag(x*)C"|)|iVI, 1}. 

Let y (1) , be as in Theorem 3.5 with e = e/2. Let £o = e/c2 and define 
i) eo (x,y) = $(x,y) + Eoy* . Define control process Y G A{q) with the corre- 
sponding state process Q [defined by the right-hand side of (2.38)] by the 
following equations. For n = 0, 1, . . . ,po, 

(3.15) Y(n0) - Y{n9-) = $ eo {Q{n9-),dY^\n)), 
and 

(3.16) Y(t + n9)-Y(n9)=t(Q(n9) + ((- + n9)-((n9))(t), te[O,0), 

with the conventions that for n = po, [0,0) is replaced by [0, oo) and for 
n = 0, Q(n9—) = q. The control Y evolves in a similar manner to Y^ at all 
time points excepting nO, n = 0, 1, . . . ,po- Since y* G T, for every q$ G R\ 
and y G R 3 , qo + Rd £0 {qQ,y) > 0. This, along with the definition of the 
map r (see below Assumption 2.15), ensures that Q is nonnegative over 
the time intervals (nO, (n + 1)9) and Q(n9) > 0, for n = 0,1, . . . ,pq. Further- 
more, (2.44) and the property Ky* > (see definition of T in Corollary 3.3) 
ensure that KY is nondecreasing and nonnegative. Thus, the process de- 
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fined by relations (3.15) and (3.16) is indeed an element of A(q). The strict 
positivity of Q at time instants n6, n<po will be exploited in the weak con- 
vergence analysis of Section 4 [see equation (4.61) and also below (4.44)]. 

Theorem 3.8. The process Y defined above is e- optimal for the BCP 
with initial value q. 

Proof. Since Y^' is e/2 optimal, in view of Lemma 3.7, it suffices to 
show that 

(3-17) |y(D_y L)T <_£_. 

For this we will introduce a collection of J-dimensional processes Yr n \ , n = 

0, 1, . . . ,p + 1, such that Y (0) = and % 0+ i) = Y {1) ■ These processes are 
only used in the current proof and do not appear elsewhere in this work. 

Define, recursively, for n = 0, 1, . . . , (po + 1), processes Yr n \ with corre- 
sponding state processes Q(n)j as follows: 

(Q ( o),%)) = (Q {1) ,^ (1) ), 

and for n > 0, t > 0, 

Q(n+1) (t) = Q(n)(t)l [O , n 0) (t) +T(H^)(t - n6)l [nfl)0o) (t) , 

Y {n+1) (t) = Y [n) (t) 1 M (t) + [Y (n) (n(9 - ) + tf eo (Q (ft) (n0-) , G>Y (n) ) 

+ r(ffW)(*-n<?)]l [nfl)00) (t), 

where for all t > 0, 

# (n) (*) = Q(„)(nfl-) + a? eo (Q (n) (n0-),0y o (n)) + C(< + ™0) " CM) 

+ i?[F(t + n6»)-F(n6l)]. 
Note that for t £ [n9, oo), 

%)(t) = Y( n) (n0-) + 1 ?(Q (n) (ne-),ay W) + f (ffW)(t - n0), 
where for all £ > 0, 

# (n) (*) = Q in) (ne-) + R^Q in) (ne-),dY (n)) + C(t + n9)-C(ne) 

+ R[Y{t + n6)-Y{n6)]. 
Using the Lipschitz property of f, it follows that, for t > 0, 

|%+i)(t) - y (n) (t)| < e (l + L\ dmg{x*)C'\)\Ry*\. 
Thus, for t > 0, 

|% 0+1) (t) - y (0) (t)| < (po + l)eo(l + L\ dmg(x*)C'\)\Ry*\ < 
The result follows on noting that Ym\ = Y^ 1 ' and Y( po+ i) = Y . □ 
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3.1. Proof of Theorem 3.5. Throughout this section we fix q G W, and 
e G (0, oo). We begin with some preparatory results. Let .4,0 be the class of all 
J-dimensional adapted processes Y given on some filtered probability space 
such that U = KY is nondecreasing, U(0) > and q + RY (0) > 0. Note that 
A(q) C .Ao- Also, a given Y G „4o is in A(q) if and only if (2.38) is satisfied. 

Given an adapted process Yq, on some system <£, with sample paths in T> 3 , 
and satisfying q + RY(0) > 0, we will denote the process Y, defined by 

(3.18) Y = Yq + T(( + RY ), 
as T(Yq). We claim that 

(3.19) if Y G Ao, then Y = T(Y ) G A(q). 

Indeed, Q = ( + RY = F(( + RY ) > 0. Also, KY = K% + KY (£ + RY ). 
Since Yq G Aq, KYq is nondecreasing and KYq(0) > 0. Also, for x G V l + , 
KY{x) = K diag(x*)C'T(x), which is a nonnegative and nondecreasing func- 
tion since Y(x) has these properties and the matrix 



Kdiag(x*)C 



Bx* n 




c 



has nonnegative entries. Combining these observations, we see that the pro- 
cess Y = T(Yq) satisfies (2.38) and (2.39). The claim follows. 

Next, from the Lipschitz property of T it follows that there is a L G (1, oo) 

such that, if 7,, (i) do,i = l,2, then for all T > 0, 

(3.20) |T(y (1) ) - T(y« )| OC)T < Z|y (1) - Y^\^ T . 

In what follows, we will denote a{X s : < s < i] by . 

Theorem 3.9. Let Y G A(q) be a {J- '^-adapted process with a.s. con- 
tinuous paths. Suppose further that for some m > 0, 

(3.21) E[|Y|™J<oo forallt>0. 

Then for any e\,T G (0,oo), there are rj,M G (0,oo), po G N and a Y^ 1 ' G 

A(q) such that Y^> = T(Y (1) ) for some Y (1) G Ao, that is, {F?}-adapted 
and satisfies (i) and (ii) of Theorem 3.5 with 9 = T/pq and 

(3.22) E[\Y-Y W\Z, T ]<e 1 . 

Proof. The construction of Y^ proceeds by defining, successively, sim- 
pler approximations of Y, denoted as Y^\Y^ 2 \ Y^ ,Y^\ The process Y^ 
is given in terms of a sequence {Y n } of J-dimensional processes, whereas 
the processes Y^ and Y^ are given in terms of one parameter families 
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of J-dimensional processes {Y(9, -),9 > 0}, {Y*(M, -),M > 0}, respectively. 
All the processes Y«, i = 1,2,3,4, and {Y n }, {Y(9,-)}, {Y*(M,-)} are only 
used in this proof and do not appear elsewhere in the paper. 

Fix ei,T G (0,oo). Define Y n (t) = n jj_ 1/n) + Y(s)ds,n > l,t > 0. Note 
that for all t,t' G [0,T], 

|F n (t)-Y n (0|<2n|t-t / ||F| 0OiT . 
Hence, by (3.21), we have, for each n, 

/O 10\ TW Y n (t) — Y n (t ) 

(3.23) E sup ■ < oo. 



sup 

■*,i'6[0,T] 



t-t' 



Note that for t G [l/n,T], |y n (t) - y(t)| < ro£(l/n) and for t G [0,1/n], 
|y n (t) - y(t)| < 2|y| 00jl/n . Since Y is continuous and Y(0) = 0, ro^(l/n) + 
2|y|oo,i/n ~~ * a - s - Combining this with (3.21) and the estimate \Y n — y|oo,T < 
2|y|oo,T, we now have that, for some no G N, Y^ = Y no satisfies 

(3.24) E\Y^-Y\Z tT <i , 
with eo =£i/4. Also, 

y^-y 1 ^') 



(3.25) 



E 



sup 

.*,t'e[o,T] 



t-t' 



= Ci<oo. 



Note that 

(3.26) y( 1 )(0) = and yWd . 

Given po G N and 9 = T/po, define 

t 



Y(9,t) = Y 1 



l [0 ,T)(t)+Y\T)l [T>oo) (t). 



Fix po large enough so that 9 < (§^) 1/m and set Y^(t) = Y(9,t). Then, 
from (3.25) we have 

E[\YW(t)-YW(t)C, T ] 



E 



(3.27) 



max < sup 

.ra=0,...,p o -l ^te[ne,(n+l)6) > 



<9 m E 



sup 



sup 



|yW(t)-yW(t')| V 



_n=0,l,...,po-l U,t'e[n6»,(n+l)e) I* ~~ M 

<0 m Ci <e - 
From (3.26) we have 

(3.28) y( 2 )(0) = and y (2) G A- 
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For let \x] denote the smallest integer upper bound for x. For x G K J , 

-l/m 

let \x] = (\xi~\ , . . . , \xj] y. Fix r\ < £ ° _ and, with convention Y^ 2 \— 9) = 0, 

po V " 

define for i > 

Lmt/6»j 

n=0 

where for y G 2? J , dy(n) denotes y(n0) — y((n — 1)9). Note that for t <T, 

[t/\T/e\ 

y(2)( t)= £ 3Y( 2 )(n). 

n=0 

Observing that for x G M J , |x — |_a^ / T7J 77 1 < rjyJ and recalling that T = p$9, 
we have that 

(3.29) E[|F( 3 ) - Y^Ct] < (poriVj)™ < e . 

Note that if y G R 3 satisfies Ky > 0, then yj < for all j = B + 1, . . . , J 
and, consequently, for such j, \yj~\ < 0. Combining this with the fact that A 
has nonnegative entries, we see that K\y\ > 0. From this observation, along 
with (3.28), we have 

(3.30) y( 3 )(0) = and y (3) d . 

The process constructed above is constant on [n9, (n + 1)9) and the 
jumps dY^ (n) take value in the lattice {krj : k G Z}, for n = 0, . . . ,po- Also, 
y(3)(t) = Y^(9 Po ) = y( 3 )(T) for f > T. 
For fixed M G (0,oo), define 

Y*(M,t)=J29Y {3) (n)I {ldY(3){n)l ^ I} , t>0. 

n=0 



Then there exists Ci G (0, 00) such that, for all M > 0, 

m T 

(|9y(3)(n)|>M)J 



P0 

(3.31) E[|F*(M, •) - r (3) |™ >T ] < C 2 ^E[|ay (3) (n)r/ ( 

n=0 



Also, for some C 3 G (0,oo), we have from (3.21), (3.24), (3.27) and (3.29) 
that, for n = 0, 1, . . . ,po, 

n\dY^(n)\ m ] < C 3 (E[\Y\™ jT ] + 1) < 00. 

Fix M > such that the right-hand side of (3.31) is bounded by Eq. Setting 
y(4) = y*(M, •), we now have that 

(3.32) n\Y^-YW\™ T ]<i . 
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Also, 

(3.33) y( 4 )(0)=0 and € A . 

Combining (3.21), (3.24), (3.27), (3.29) and (3.32), we now have that y o (1) = 
satisfies (3.22) as well as (i) and (ii) of Theorem 3.5. This completes 
the proof. □ 

Lemma 3.10. For each e\ > there exists an ei-optimal Y € A(q), 
which is {J- -adapted, continuous a.s., and satisfies 

(3.34) limsupe- 7T E|y|™ T = for every m > 0. 

T->oo 

Proof. Fix e\ > and let w = Aq. Applying Theorem 2.1(iv) of [2], we 
have that Jq(w) = inf Jq(w, U), where the infimum is taken over all {J^ }- 
adapted controls U. Hence, using (3.10), we conclude that there is an N- 
dimensional {J-^j-adapted process U for which (3.7) holds and 

(3.35) J*(q) = J *H > J (w, U)-e x . 

From Lemma 4.7 of [2] and following the construction of Proposition 3.3 
of [10] [cf. (12) and (14) of that paper], we can assume without loss of 
generality that U has continuous sample paths and for all m > 0, 

limsupe- 7T E|GC/|™ T = 0. 

T->oo 

Hence, using properties of the G matrix (see Assumption 2.6 ), we have that 
limsupe- 7T E|C/(T)| m < C - m limsupe- 7T E|GC/|S T = 

T->oo T->oo 

(3.36) 

for all m > 0. 

We will now use a construction given in the proof of Theorem 1 of [20] . This 
construction shows that there is a J x N matrix F\ and a J x I matrix i*2 
such that letting 

(3.37) Y(t)=F 1 U(t)+F 2 (q*(W(t))-q-X(t)), t > 0, 

we have that Y € A{q) and Jo(w, U) = J(q, Y). We refer the reader to equa- 
tions (35) and (36) of [20] for definitions and constructions of these matrices. 
From (3.35) we now have that Y is an ei-optimal control, has continuous 
sample paths a.s. and is {J 7 * }-adapted. Finally from (3.37), we have that 
for some C2 G (0, 00), 

®\YC,T < C 2 (l + nU(T)\ m + T m + E\q*(W)\™ >T ). 
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By combining (3.3) and (3.7), the fourth term on the right-hand side can 
be bounded above by C 3 (l + T m + E,\U(T)\ m ) for some C 3 > 0. The result 
then follows on using (3.36). □ 

The following construction will be used in the proof of Theorem 3.5. 

Lemma 3.11. Fix Y £ A{q) such that J{q,Y) < oo and (3.34) holds. 
For T > 0, let Y(f(t) = Y(tf\T), t > 0, and Y T = T(Y T ). Then given e\ > 0, 
there exists T E (0, oo) such that \ J(q,Y) — J(q,Y T )\ < e± and (3.34) holds 
with Y replaced by Y T . 

Proof. Since J(q,Y) < oo, we have that 

poo r 

L(T,Y) = E e-^h-Q(t)dt + E e'^p- dU(t) ^0 asT^oo, 

where Q is the state process corresponding to Y and U = KY . Choose T\ 
large enough so that 

(3.38) L(T,Y) < £i/2 for T>T\. 

Using the Lipschitz property of T and T (see Assumption 2.15), we can find 
Ci E (0, oo) such that, for all T > 0, 

\Q T (t)-Q{T)\ + \Y T {t)-Y{T)\ 

(3 39) 

<d sup |C(s)-C(T)|, t>T, 

T<s<t 

where Q T is the state process corresponding to Y T . Thus, for some C2 £ 
(0,oo), 



E 



f'OO L f'CXJ 

/ e-^h-Q T (t)dt < -e-^ T E\Q(T)\ + C? / e-^(l + t)dt, 
Jt 7 Jt 



where h = maxjgi hi . Using (3.34), we can now choose T2 large enough so 
that 

POO 

(3.40) E J e" 7 */i • Q T (t) dt < ei/4 for all T > T 2 . 

Next, letting U T = KY T , we have from (3.39) that for some C3 £ (0,oo), 

E\U T {t) - U(T)\ < C 3 (l + (i - T)) for < T < t. 
Integration by parts now yields that for some T3 > 0, 

(3.41) E J e" yT p-dU T (t)<e 1 /4 for T>T 3 . 

Combining the estimates in (3.38), (3.40) and (3.41), we now have that for 
all T > max{Ti, T 2 ,T 3 }, 

I J(q, Y) - J(q, Y T )\ < L(T, Y) + L(T, Y T ) < e x /2 + £l /4 + s 1 /A = e 1 . 



n\Y T - ^ (1) Ioo,t] < m\Y T - y (1) L lT ] < 



32 A. BUDHIRAJA AND A. P. GHOSH 

Finally, the fact that (3.34) holds with Y replaced by Y T is an immediate 
consequence of (3.39). □ 

We can now complete the proof of Theorem 3.5. 

Proof of Theorem 3.5. Using Lemma 3.10, one can find Y 6 A(q) 
which is e/5-optimal, has continuous paths a.s., is {J^j-adapted and sat- 
isfies (3.34). Using Lemma 3.11, we can find T £ (0,oo) such that Y T = 
T(Y(- A T)) is 2e/5-optimal and (3.34) holds with Y replaced by Y T . We 
will apply Theorem 3.9, with m = 1, Y replaced with Y T , s\ replaced 
by e/(5ci-L) and denote the corresponding processes obtained from The- 
orem 3.9, once again by Y^ and Yq 1 ^. In particular, Y^ 6 -A{q) is such 

that Y^ = Y(Y ), where Yq is {J^j-adapted, satisfies (i) and (ii) of The- 
orem 3.5, for some rj, M, 8 G (0, oo) and po G N, and (3.22) holds with m = 1, 
Y replaced by Y T and where L is as in (3.20). Then 

e 

5ci 

Thus, from Lemma 3.7, Y^ is 3e/5-optimal. 

Processes Y^\ Y^ as in the statement of Theorem 3.5 will be constructed 
by modifying the processes F (1) ,yW above (but denoted once more by the 
same symbols), by constructing successive approximations (Yq ,Y^) and 

(Yq , Y"W). These approximations are only used in the current proof and 
do not appear elsewhere in the paper. 

Consider « > such that 9/k G N. Let = a{X K {n)}, 0( n ) = a{X(s) : 
s<n6},g = a{X{s):s>0},n£-N,K>0. Since t G (n) as /clO and Y (1) 
is {J^j-adapted, we have for each fixed <j G S n M , 

p[dy (1) (n) = q \gW] -> p[ay (1) (n) = q \gW] 

(3.42) 

= l r „v>(i), s , a.e., as kIO. 
Note that for fixed ? G 5]^ and n = 1, . . . ,po> 

p[ay (1) (n) = ? |gW]=K, ? (^H), 

for some measurable map p* f rM" - - 701 — > [0, 1] satisfying 
(3.43) for all x G R lnjo ^ K,^) = 1- 

Using Lemma A.l in the Appendix, we can construct, by suitably aug- 

(k) 

menting the filtered probability space, an adapted process Yq such that 
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y W (0) = 0, y W (t) = Y^\\t/6\6) and dY^\n) = Y^\n6) - y w ((n - 
1)6) satisfies 

(3.44) P[«9y (K) (n) = s\G V yr 1 ] = 1 < n < Po , ? E , 

where = °{dY \ < n}. Note that if / is a real bounded continuous 

map on C 1 , then by successive conditioning and (3.42), as k|0, 

Cpo \ / po \ 

/wnv« w =, } J = n^(^(n))j 

^ E (/wflW(n)=.})' 



n=l 



for any fixed (?i, . . . , ? Po ). Thus, in particular, 

(3.45) (l,y {K) )^(l,y (1) ) + 

Define = T(Y (k) ). Then y(«) E .%) and satisfies (i) and (ii) of The- 
orem 3.5. Also, (3.45) along with the Lipschitz property of the Skorohod 
map yields that J{q,Y^) J(g,Y (1) ) as k J, 0. Fix k sufficiently small so 

that Y( K ) is 4e/5-optimal and, suppressing k, denote Y^ by Yq, Y^ by Y 
and the generating kernels by p n ,? • 

Finally, in order to ensure property (iii) in the theorem, we mollify the 
kernels p n ^ as follows. For 7 > 0, and n = 1, . . . ,po, define 

(3.46) pl s (x)= pn^x + z^i^iz^dzj), xeR nj0 \^Sl, 

where (jf 1 is the density function of an I-dimensional Normal random variable 
with mean and variance 7/. Note that the map x ^ pZ,,q{x) is continuous 
for every 7,n, <j and (3.43) is satisfied with p* q (x) replaced with pZ,<;(x). 
From continuity of the maps p™,?, we can find measurable maps (suppressing 
dependence on 7 in notation) w n : IR 1 ™- 70 x [0, 1] — > Sjj, n = 1, . . . ,po, such 
that zu n (-,t) is continuous, at every x EM 1 ™- 70 , for a.e. t in [0,1], and if U is 
a Uniform random variable on [0, 1] , then 

F(w n (x,U)=^)=pZ, q (x), x E R Iryo , <; E S V M . 

Let {U n } be an i.i.d sequence of Uniform random variables, that is, inde- 
pendent of X. Now construct Y^ such that Yq 7 \o) = 0, 

dY^\n) = Y^\n9) - Y^\(n - 1)6) = w n (X K (n),U n ), n > 1 

and Y^\t)=Y^\6\t/6\). Note that 

F[dY^(n) = ,\G V y^ 1 ] = j^(*»), n > 1,, E 
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Since p7i,s — > p n ,<; pointwise, as 7 — > 0, we have for every real bounded map / 
on C 1 , 

CPO \ / PO \ 

^E[f(X)f[p n ,, n {X«(n))) 



n=l 
Po 



{ar (n)=?n} I ' 

n=l / 



for every (ft, . . . ,? po ), as 7 ->■ 0. Thus, (X, Y^) => (X, Y ), as 7 ->■ 0. Let 
y( 7 ) = T(y (7) ). Then G .A(g) and the above weak convergence and, once 
more, the Lipschitz property of the Skorohod map yields that J(q,Y™)-> 
J(q, Y) as 7 J. 0. Recall that Y is 4e/5-optimal. We now choose 7 sufficiently 

small so that Y^ is e-optimal. By construction, Yq ^ = Yq and Y^ 1 ) = Y^ 
satisfy all the properties stated in the theorem. □ 

4. Asymptotically near-optimal controls for SPN. The goal of this sec- 
tion is to prove Theorem 2.17. Fix q G K]j_ and e G (0, 1). Let Y G A(q) be 
the e-optimal control introduced above Theorem 3.8. Fix q r G Nq, r > 
such that q r — > q, as r — > 00. Section 4.1 below gives the construction of 
the sequence of policies {T r }, T r G A r {q r ), such that J r (q r ,T r ) — > J(q,Y), 
yielding the proof of Theorem 2.17. The latter convergence of costs is proved 
in Section 4.2. The main ingredient in this proof is Theorem 4.5 whose proof 
is given in Section 4.3. For the rest of this section Y as in Theorem 3.8 and 
parameters T,rj,M,po,jo,9,K that specify Y shall be fixed. In addition, let 
p G (0, 00) and tq > 1 be such that 

(4.1) p minx* > M(#n p + 1) and r 6 > p, 

lx*?o J 

where i?np is as in (3.13). 

4.1. Construction of the policy sequence. We will only specify a T r £A r (q r ) 
for r > tq and so henceforth, without loss of generality, we assume r > tq. 

In this section, since r > r$ will be fixed, the superscript r will frequently 
be suppressed from the notation. The following additional notation will be 
used. For n < po, define a(n) = nr 2 6, b(n) = nr 2 9 + rp, 2(n) = [a(n),a(n + 
1)), T\{n) = [a(n),b(n)) and 22(71) = [b(n),a(n + l)), where we set a(po + l) = 
00. Note 2(n) = 2i(n) U 22(71), n < pq. 

Recall m introduced in Assumption 2.13. Fix k G (0, 1) such that 

(4.2) k(l + m) -2 = v> 1. 
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o(0)=0 a(l) = r 2 8 a(n)=nr 2 9 a(n + 1) = (n + l)r 2 6 a( Po ) = p B r 2 = r 2 T a(p + 1) ■ 




m2 many A-intervals 

A A 
-I 1 u 




iiUM -</(/') + / V^a(n + 1) = (ra + l)r 2( - 
a(ra) + (mi + l)A 6(n) + m 2 A 6(ra) + (m 2 + 1)A 



works on 
act. ■ 



Fig. 4. T/ie figure above shows the behavior, under of the kth server, that is, re- 

sponsible for the nk activities (ji, . . . ,j„ k ). The actual policy T is given as a certain 
modification of that ensures feasibility and a strict positivity property. 



Fix d\ G (0, oo) such that d\ > sup r {/3J : j G J} + 1. Define 
(4.3) A r = r k , e r ( S ) = dir k l (Un . po x 2{n)) ( S ) for a > 0. 
Also define p r : [0,oo) — > [0,oo) as 
s, 



p r (s) = 



b(n) + 



b(n) 



A 



A. 



if sGXi(n),n = 0,l,...,p , 
if se^(n),n = 0,l,...,po. 



Thus, if s € Tiivi) for some n, G r (s) = diA r and p r (s) equals the left end 
point of the A-subinterval in which s falls. Otherwise, if s 6 Xi (ra) for some n, 
G r (s) = and p r (s) = s. 

Recall the probability space (f2, J 7 , P) introduced in Section 2 which sup- 
ports all the random variables and stochastic processes introduced therein. 
Let {tii :i £ N} be a sequence of Uniform random variables on [0, 1] on this 
probability space (constructed by augmenting the space if needed) , indepen- 
dent of the o"-field T r defined in (2.27). This sequence will be used in the 
construction of the control policy. 

The policy T = T r is constructed recursively over time intervals X(n), n = 
0,...,po, as follows. We will describe the effect of the policy on the A;th 
server (for each i;£l) at every time-instant s > (see Figure 4). Recall 
the set J(/c) introduced in Section 2. Let for k EK, ji< j%< ■•• < jn k be 
the ordered elements of J(/c) (the set of activities that the kth server can 
perform) . 
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Step 1: [T r (s) when sGl(O)]. Let m = [^J . Recall that F(0) = e y* 
Write ^° = e y*- For k G K, let 



(4.4) 



V 



l,fc,0 • 



A, V 



Note that, by our choice of p, V^' fc '° > if ji is a basic activity. Also, if ji is 

nonbasic, then x* ; = 0, but since Kv° > 0, we have 1/° < 0. Thus, Vj' fe '° > 
for every l,k. Also, from Assumption 2.3 [see (2.14*)] and recalling that 
Av° > 0, we get 

(4.5) ^V^ = A(l-i(^ ) fc )<A, £vf'° = A, kK. 
z=i \ P J l=1 

Let mi = m'J = |_^J and m 2 = rri^ = [ r ^~ rp J • Define, for each k G K, and 
Z = l,...,n fc , 

mi —1 

l f 7n,o(s), sGXi(0), 



2,fc,0 • 



4 A > 



1, ... ,71ft. 



*foo=< 



m=0 
m 2 — 1 



^ l^m,o(s), sGl 2 (0), 



k m=0 

where for rrij = 0, . . . , wij — 1, i = 1, 2, Z = 1, . . . , n^, 



(4.6) 



(4.7) 



mi,0 



8 



i-1 J \ 

a(0) + mi A + V}' fc '°, o(0) + mi A + vj ,fc '° 

i=0 i=0 / 

6(0) + m 2 A + V • ' fc '°, &(0) + m 2 A + ^ V 4 2 ' fc '° 

i=0 i=0 / 



and, by convention, Vq^' = 0, z = 1, 2. Since {S(k), k G K} gives a partition 
of J, the above defines the J-dimensional process T«(s) far se[a(0),a(l)). 
We set 



TW(«)= [ S f^{s)ds, sGJ(0). 
Vo 



Next, define V(s) = V r (s) = (Q r (s),X r (s),T r (s)) for s G 1(0) by the system 
of equations below: 

<7( a ) = q r + X r (s) + r(5ls - (C - P 7 ) diag(r 2 0£)T r (s/r 2 )) 

+ R(x*s-T r (s)), 
(4-8) . 

X r (s) = rl r (s/r 2 ), X r defined by (2.23), 



[ {Q; i(j) («-)>o} 1 {Q; iU) (P''(«))>e-(«)} dTj' (1) (u), 3 G J, 
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where (as introduced in Section 2), cr\(j) denotes the index of the buffer that 
the jth activity is associated with. The above construction can be interpreted 
as follows. The policy T "attempts" to implement over Xi(0), unless 
the corresponding buffer is empty (in which case, it idles). Over ^2(0), T has 
a similar behavior, but, in addition, it idles when the corresponding buffer 
does not have at least — many jobs at the beginning of the A-sub intervals. 
Processing jobs only when there is a "safety-stock" at each buffer at the 
beginning of the A-subinterval ensures that with "high probability" either 
all the activities associated with the given buffer receive nominal time effort 
over the interval or all of them receive zero effort. This, in particular, makes 
sure that the idling processes associated with the policy are consistent with 
the reflection terms for the Skorohod map associated with the constraint ma- 
trix D [see (4.21)]. This property will be exploited in the weak convergence 
arguments of Section 4.2 [see, e.g., arguments below (4.45)]. 

Let T r be defined by (2.9). By construction, T r (0) = 0, I r (0) = and 

Q r (s)>0, T r (s)>0, F(s)>0 

(4.9) 

for s G X(0) and T r ,F are nondecreasing on Z(0). 

This completes the construction of the policy and the associated processes 
on J(0). 

Step 2: [T r (s) for s £l(u), for 1 < n < po\. Suppose now that the pro- 
cess V(s) has been defined for s G [a(0),a(n)), where 1 < n < pq. We now de- 
scribe the construction over the interval 1{n) = [a(n),a(n + l)). Let x K ' r (n) = 
{X t {Ik) : I = 1, . . . , njo}. Recall that njQK = nO. Define 

(4.10) v r > n = uj n (x K ' r (n),Un), v r > n = ^ £0 (Q r {n6-),u r ' n ), 

where, as in Section 2, Q r (t) = Q r (r 2 t)/r, t>0. We will suppress k and r 
from the notation and write {x K,r {n),i> r,n ,v r ' n ) = (x(n), v n , u n ). For each 
k G K, define 



(4.11) V ]' fe ' n =(4-^jA, V 2 e ' k ' n = x* e A, £ = l,...,n k . 

As before, (4.5) (with V^ fc '° replaced by V^' fc ' n ) is satisfied. Define, for each 
k G K, and 1 = 1,..., rtfe, 

mi — 1 

1 ep n ( s ), s€Xi(n), 



(4.12) fl(s) 



m=0 
m-2 — \ 

} j lgm,n(s), sGX 2 (n), 

m=0 



where for rhi = 0, . . . ,rrii — 1, i = 1,2, 1 = 1,... ,n k , £, u ,£, 2 ' are defined 
by the right-hand side of (4.6) and (4.7) respectively, with (a(0), 6(0), V^' fc '°, 
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Vf fc '°) replaced by (a(n), 6(n), Vj'*' n , V 4 2 ' fc ' n ) We set 

r 1 (s)=T(a(n)) + / t 1 (s)d8, s G Z(n) 

J a(n) 

and define V(s) for s G X(n) by the system of equations in (4.8). 

The above recursive procedure gives a construction for the process V r (s) = 
(Q r (s),X r (s),T r (s)) for all s G [0, oo) . 

The policy constructed above clearly satisfies parts (i), (ii) and (iii) of 
Definition 2.7. In fact, it also satisfies part (iv) of the definition and, conse- 
quently, we have the following result. The proof is given in the Appendix. 

Proposition 4.1. For all r > r , T r G A r (q r ). 

4.2. Convergence of costs. In this section we prove Theorem 2.17 by 
showing that, with {T r } as in Section 4.1 and Y as introduced above The- 
orem 3.8, J r (q r ,T r ) — > J(q,Y), as r — > oo. We begin with an elementary 
lemma. 

Lemma 4.2. Let g n G V 1 , n > 1, g G C 1 be such that g n — > g u.o.c. as 
n oo. Let e n G (0, oo), n > 1, be such that e n — > as n — > oo. Let B n , B 
be I x I matrices such that B 11 — > B as n — > oo. Suppose f n , h n G f 1 and 
7 n G P 1 sate/y /or all t>0: 

(i) H*)>0, 7 n W>0, 

(ii) /^^"(il + W^^^^l^^^dvel and 

(iii) |7 n (i)-*l 

TTien (f n ,g n ,h n ,^ n ) is precompact in 2? 3I+1 and any Zimzi point (f,g,h,j) 
satisfies for all t>0, j(t) = t; f(t) = g{t) + Bh(t); h(t) = Jj ^ /i(s) ds, where 

h: [0, oo) — > [0, l] 1 is a measurable map such that 

/ 1 {f l (s)>0}hi{s)ds = forallt>0. 
J[o,t] 

The proof of the above lemma is given in the Appendix. Recall the def- 
initions of various scaled processes given in (2.21)-(2.25), and that i(t) = t 
for t > 0. In addition, we define f r ^{t) = T r ^(r 2 t)/r 2 , T^t) = T r ' 1 (r 2 t)/r, 
r>0, f >0. 

PROPOSITION 4.3. As r ^ oo, T r — > x*i, u.o.c. in probability. 

Proof. From the definition of T r ' 1 [see (4.12)], and the observation that 
the interval £ / m,n has length x* A [see (4.5) and (4.7)], we have that over 
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each interval [mA, (m + 1)A), that is, contained in l2( n ) for some n <po, 
the Lebesgue measure of the time instants s such that Tj (s) = 1 equals x*A, 
j € J. This is equivalent to the statement that 

/ d(T;>\s)-x*s) = 

J[mA,(m+l)A) 

whenever [mA, (m + 1)A) C ^(n) for some n<pQ. 

Also, noting that for j G JJ, < Tj' 1 < 1 and x* < 1, we have that for some 
Ci>0, 



/ 

J\rr 



d(T^ 1 \s) + x*s)<C 1 A 



[mA,(ra+l)A) 

(4.14) 

whenever [mA, (m, + 1)A) C X{n) for some n < pq. 
Fix j G J and t > such that r 2 t G [a(n), a(n + 1)) for some n < po- Then 

n 2 „ 

n =0 £=1 m : [ m A,(m+l)A)eX, (no) 7 [™ A At,(m+l) AAt) 

Using (4.13) and (4.14) and the fact that the number of A intervals in Ii(n) 
is bounded by rp/A, we see that 

r^-^ 1 WI<( ! ^^ + l)c 1 A. 

Thus, for some g G (0,oo), 

(4.15) sup \x*t-T r ' 1 (t)\<g/r for all r > r . 

0<t<oo 

Also, since TJ(t) < i for all £ > 0, j € J, we get from (2.23) and standard 

estimates for renewal processes (see, e.g., Lemma 3.5 of [9]) that X r = X r /r 
converges to u.o.c. in probability, as r — > oo. Combining this with Assump- 
tion 2.3 and (4.15), we have 

(4.16) Q r = C l r converges to 0, u.o.c, in probability. 
Next, define p r (s) = p r (r 2 s)/r 2 , @ r (s) = Q r {r 2 s)/r 2 and 

SJ(t) = {s G [0,t]:Ql iU) (s) = 0,or Q r ai(j) (p r (s)) < & r (s)} 

= {se[0,t]:Q: iU) (p r (s))<@ r (s)} 

(4-17) 

U {s e [0,t]:Q* l(3l (8) = 0,Q: i(j) (p r ( S )) > @ r (s)} 
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We will fix t > for the rest of the proof and suppress t from the notation 
when writing Sj' l (t), unless there is scope for confusion. Using the above 
display and (4.8), we have that 

(4.18) Tf(t)-Tj(t)= / l S v(s)dfJ'\ S ), jel,t>0. 

J[o,t] 3 

— f\r ( \ 

Using the fact that Q r {-) = ^-^- L along with (2.24) and (4.18), we can write 

(4.19) Q r {t) = C(t) + RH r (t)+RL r (t), t>0, 
where for j G J, 

H](t) = {x*t-f^(t)) + [ l s r, l (s)d(f;'\s)-x*s) 



(4.20) 



and 



where 



J[0,t] 3 

L r At)=x* [ W(s)ds = (diag(x*)C'r),(f), 
7ro.il 



£i(t) = J ^ 1 {Q r l (p r (s))<0r(s)} ds, t>0,ie I 

Since D = Rdiag(x*)C [see (2.11) and (2.43)], we have 

(4.21) Q r = C + RH r + DC . 

Next, letting mjj = L^rJ + 1, we have from the choice of d\ [see above (4.3)] 
that 

P(5}' 2 ^ 0) 

< E E nQl(kA r ) > d ir \ Q\{u) = 

k=0 iel 

for some u G [A;A r , (fc + l)A r )) 

(4.22) <I^^P(5J(7; r (A : A r ) + A r ) - S^T] (k£ r )) >d ir k ) 

k=o j'eJ 



<mHJ 



^km 



<C 2 (t + l)r 2 - k ( 1+m ), 
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for some C2 G (0, 00), where the next to last inequality makes use of As- 
sumption 2.13. Recalling [from (4.2)] that v = k(l + m) — 2 > 1, we get that 

(4.23) F(Sj' 2 ^ 0) < C 2 (t + l)r~ v -»• as r -»• 00. 

We note that the above convergence only requires that v > 0. The property 
v > 1 will, however, be needed in the proof of Proposition 4.4 [see (4.28)]. 
Next, using (4.13) and (4.14), we have for some q\ G (0, 00), 



(4.24) sup 

0<u<oo 



l s r,i d(T.'\s)-x*s) 

[0,U] 3 J 



< )• as r — )• 00, 



for all j G J. The above inequality follows from the fact that the integral can 
be written as the sum of integrals over A-subintervals: When the subinterval 
is within some X2{n), the integral is zero [using the definition of p r (s) in 
such intervals and (4.13)] and when the subinterval is within some X\{n) 
[the number of such intervals is (po + which can be bounded by Cs^£ 
for some C3], the integral is bounded by C±A/r 2 from (4.14). 

Now, combining (4.15), (4.23) and (4.24), we have that, for each j G J, 
Hj — > 0, u.o.c. in probability, as r — > 00. From (4.21), (4.16), Lemma 4.2 
and unique solvability of the Skorohod problem for (0,-D), we now have that 
(Q r ,£ r ) — > (0,0), u.o.c. in probability, as r — > 00. Thus, U converges to as 
well. The result now follows on noting that T r = x*i — H r — 17. □ 

The following proposition gives a key estimate in the proof of Theo- 
rem 2.17. 

Proposition 4.4. For some C2 G (0,oo) and v £ (l,oo), 

H\Q r \to,t + \C\lo,t + \Y r \to,t) < 02(1 + 1 3 ) for all r>r ,t>0. 

Proof. Using standard moment estimates for renewal processes (cf. 
Lemma 3.5 of [9]), one can find C\ G (0, 00) such that 

(4.25) IE|C r |L,t <Ci(l + t 2 ) for allr>r ,t>0. 

From (4.19) and (4.21), we have 

Q r (t) = C(t) + RH r {t) + RU\t) = C(t) + RH r {t) + Df(t), t > 0, 

where H r = rH r , U = rll and £ r = r£ r . We rewrite the above display as 

Q r (f(t)) = [Q r (p r (t)) - Q r {t)} + C(t) + RH r (t) + Df(t), t > 0. 

From Theorem 5.1 of [29], for some Ci G (0,oo), 

W\oo,t + \Q r \ 

(4.26) 

< C 2 (q r + \C\oc,t + |£ r W + IQ r (p r (-)) - Q r \oo,t + —) ■ 
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Also, from (4.20), (4.15) and (4.24), for all t > and r > r , 



(4.27) 



m(t)-r I uw^. 



E 



(4.28) 



r sup / l s r,2(s)dT^(s)) <r v C 3 {t s + l)r' 

0<u<tJ[0,u] i 



'[OA ~ 3 

Next, using (4.23), we get, for some C3 G (0,oo), 

v/\2 



<C 3 (t 3 + l). 



Finally, for some C4 € (l,oo), for all a> 1, 
P(IQ r (p r (-))-Q r (-)loo,t>a) 

(4.29) 



<£( E p(^)^) + £p(^£)). 

Using moment estimates for renewal process once more (Lemma 3.5 of [9]), 
we can find C5 € (0, 00) such that 

n\Q r (p r (-))-Q r (-)\oo,t>a)< r 



< 



A r (ar - C 5 A r ) 2 
1 C 5 r 2 t 



a 2 (r-C 5 A r /a) 2 ' 
Thus, there is an r\ G (ro,oo) and C§ G (0,oo) such that, for all r > r\, 



(4.30) 



P(|Q r (p r (-)) - Q r {-)\oo,t >a)< \c 6 (t + 1 



This shows that for some v\ G (l,oo) and C7 G (0,oo), 

(4.31) n& {?(■)) - Q r (-)\2,t < cva + 0, « > 0. 

The result now follows on using (4.25), (4.27), (4.28) and (4.31) in (4.26) 
and observing that Y r = H r + di&g(x*)C l r . □ 

In preparation for the proof of Theorem 2.17, we introduce the following 
notation. For n = 0, . . . ,po — 1, we define processes q r,n , z r,n with paths in T)\ 
and Cq , respectively, as 



(Q r (n9 + p/r),0), t€[0,p/r) 
(Q r (t + n9),Y r (t + n9)-Y r (n0 + p/r)), te[p/r,0]. 



We denote by cf ,Po , z r ' Po the processes with paths in P 1 and C J , respectively, 
defined by the right-hand side in the display above, by replacing n by po 
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and [p/r, 9] with [p/r, oo). Then 

q r = (q r >*,q r 'P°)£V p g o1 xV 1 a.s., 
where q r '* = (<f'°, ■ ■ ■ , q r,po ~ 1 ). Similarly, define the process z r with paths in 

C p ° 3 x C 3 . Recall v r ' n ,v r > n introduced in (4.10). Denote P r = (p r >°, P r,1 5 ■ • ■ , 
u r ' p °) and v r = (v r, °, is 7 "' 1 , . . . , v r,po ), where we set P r '° = and u r >° = eoy* . 

Next, for n = 0,...,po — 1, define processes q^ n \z^ with paths in C\ 
and , respectively, as 

f Jn) (t) in)(t))= ((Q(t + n9),Y(t + n9)-Y(ne)), te[0,0), 
{q U ' UJ l(Q((n + l)e-),y((n + l)fl-)-y(ne)), i = 0. 
Also, define q^ Po \ z^ Po ^ by the first line of the above display by replacing 9 by 
oo. Then q = (?*, ? {po) ) G C Po1 x C 1 , a.s., where q* = (f(°>, . . . , q^°~^). Simi- 
larly, define the process z with paths in C Pl)J x C J . Also, let for n = 1, . . . ,po, 
= <9Y (1) (n) = Y (1) (n<9) - Y (1) ((n - 1)0), where y o (1) is as above (3.15), 
and i» = F(n0) - F(n0-). Then 

(4.32) = s7 n (x»,Z4), = 6o (Q(n0-),pW). 

Define P = (p(°),p( 1 ),...,p( po )) and i/ = i/W, . . . , where P<°) = 

and = e y*. Then P r ,P E (5^)®( po+1 ) and j/> E Next, let 

^• n = ;T( n + p/r), ^ n) =Y(n9), n = 0, 1, . . . ,p . 

Then V r = (v r >°, . . . , y Q = { v f\. . . , u^) E M J ^+i). Let 

H = D 1 x (5^)® (po+1) x (E J ( po+1 )) x (M J to' +1 )) x (Vf l x P 1 ) x (C p ° 3 xC 3 ). 

Note that J T = (£ r , P r , ^5' ? r > r — 1 anc ^ <^ = (C> ^Oj (7j -2) are E- 
valued random variables. The following is the main step in the proof of 
Theorem 2.17. 

Theorem 4.5. As r — > 00, J r =>• J. 

Proof of the above theorem is given in the next subsection. Using Theo- 
rem 4.5, the proof of Theorem 2.17 is now completed as follows. 

Proof of Theorem 2.17. From proposition and integration by parts, 

po 



r(q r ,T r ) = y E / e-*(h-Q r (t) + >yp-U r (t))dt 

(4.33) + E / e-^(h-Q r (t) + -f P -U r (t))dt 

J[a r {n)/r 2 ,b r {n)/r 2 ) 
Pa n 

= VE / e-^(h-Q r (t) + -f P -U r (t))dt + e r 

J\b r (n)/r 2 ,a r (n+l)/r 2 ) 
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where, using Proposition 4.4 and the observation that a r (n + l)/r 2 — b r {n)/ 
r 2 < p/r —7- 0, we have that e r — > as r — > oo. From Theorem 4.5, as r — > oo, 
q r =^ q. Combining this with Proposition 4.4, we get for every n = 0, . . . ,po, 



lim E / e-^h-Q r (t)dt 

r ^°° J[b r (n)/r 2 ,a r (n+l)/r 2 ) 

= lim E / e _7 */t-Q r (t)dt 

r ->°° J{n0+p/r,(n+l)e) 

(4.34) = lim E f e'^h ■ q r > n (t - nO) dt 

r ^°° J[n6,(n+1)6) 

= E I e'^h- q ( - n) (t-n9)dt 

J[nd,(n+1)0) 

= E f e~^h-Q(t)dt, 

J[n6,{n+1)6) 

where, by convention, [n9, (n + 1)9) = \po0, oo) when n = pq. Next, for t G 
[n9 + p/r, (n + 1)9), n = 0, . . . ,p , 

1V . tj r (t) = 7p • K(Y r (t) - Y r (n9 + p/r)) + 7 p • KY r (n9 + p/r) 

(4.35) 

= 7 p • Kz r > n (t - n&) + 7 p • Kvl> n . 
From Theorem 4.5, as r — > oo, 

7 p • 2f* p ' n (.) + 7 p • Kv£ n 1V ■ Kz {n) {-) + jp ■ AvJ n) 

in Cg. Combining this with (4.35) and Proposition 4.4 we now get similarly 
to (4.34), for n = 0,...,p Q , 



lim E / je- jt p-U r (t)dt 
J[b r (n)/r 2 ,a r (n+l)/r 2 ) 

= lim E [ je-^p ■ K(z r ' n (t - nO) + u!> n ) dt 

r ^°° J[n0,(n+1)9) 

= E f 1 e-^ t p-K^ n \t-ne) + u^ ) )dt. 

J\n6,(n+1)6) 



'[n9,(n+l)e) 

Note that for t G [n9, (n + 1)9), 

z {n \t - n9) + v { q ] = Y(n9) + Y(t) - Y(n9) = Y(t). 
Thus, the expression on the right-hand side of the above display equals 



E f 7e" 7 'p • U(t) dt. 

J\ne,(n+1)6) 



[n9,{n+l)e) 

The result now follows on using this observation along with (4.34) in (4.33). 

□ 
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4.3. Proof of Theorem 4-5. For j & J, n = 0, 1, . . . ,po, and w £ fi, define 

(4.36) Sy n (u) = {s£ [0, p] : Q; i{i) (nr 2 + w, w) = 0}. 
From the definition of i? eo , it follows that for some 70 > 0, 

inf min (Rv T,n )j > 70 a.e. 

r>r n=0,...,po;j'6JI 

As a consequence of this observation, we have the following result. The proof 
is given in Section 4.4. 

Proposition 4.6. For some {p r } C [0,p] such that p r -> as r -> 00, 
we have 

(4.37) P(^) -)• 1 as r -> 00, /or aZZ n = 0, 1, . . . ,p , 
where = {w£fi: (U ieJ D k, p] = 0} • 

For n = 0, 1, . . . ,po, let P r [n] = (P r, °, . . . , P r,n ). We define v r [n] , i/q [n] , q r [n] , 
z r [n] and their limiting analogues P[n], v[n], vo[n], ^M^M m a similar fash- 
ion. Set 

J r [n] = (C,v r [n],u r [n],v r [n],f[n],z r [n]), 

J[n] = (C,P[n],u[n],u [n\, q[n],z[n]). 
Then J" r '[n],J'[n] are S[n]-valued random variables, with 

E[n] =V l x (S^f (n+1) x (R 3 ^) x (M J ( n+1 )) x pj n+1)1 x cJ n+1)J , 

where we follow the usual convention for n = pQ. To prove the theorem, we 
need to show that c7 r [po] <J[Po] - in the lemma below we will in fact show, 
recursively in n, that J r [n] J[n] as r — )• 00, for each n = 0, 1, . . . ,po, which 
will complete the proof of Theorem 4.5. 

Lemma 4.7. For each n = 0, 1, . . . ,po, J r [n] =>• ^[n], as r — > 00. 

Proof. The proof will follow the following two steps: 

(i) Asr^oo, J r [0]^J[0}. 

(ii) Suppose that J r [k] =4- J[k] as r — >• 00 for A; = 0, 1, . . . , n, for some 
n<p . Then, as r -»■ 00, J" r [rj + 1] J7"[n+ 1]. 

Consider (i). Define scaled processes Q r (t) = Q r (rt)/r, Y r (t) = Y r (rt)/r. 
Processes X r , T r , T r ' 1 , £ r are defined similarly. By the functional central 
limit theorem for renewal processes and Proposition 4.3, it follows that (cf. 
Lemma 3.3 of [9]) 

(4.38) C^C 

Also, convergence of (^ r [0], v r [0]) follows trivially since v r [0] = P[0] = and 
z^ r [0] = v[0] = £qd* . Next, consider Vq[0] = Y r (p/r). From the definition of 
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the scaled processes defined above (4.36), we have that 

(4.39) ( r (t) = f + X r (t) +-[9[t - (C - P') dmg(e r 2 )f r (t)}, 

r 

and 

(4.40) Q r (t) = C(t)+RY r (t), te[0,p]. 
Also, observe that Y r can be written as 

P 



(4.41) Y r (t) = 
where, with S'J' defined in (4.36), 

(4.42) JVJ(t) 



t-T^(t))+-e y*t + N r (t), 
P 



l*r,o(s) df T j ' l {s). 
[0,t] ' 



Next, note that, for a suitable C\ E (0,oo), 

, toy 



(4.43) 



rpr,l 



p 



oo,p 



<Cx—. 

r 



Also, l-ZV^loo^ < p r + pl(^r)c. Thus, from Proposition 4.6, liV 7 " |oo,p converges 
to in probability as r — > oo, which shows that 



(4.44) 



y r - — y* 



oo,p 



X 







oo,p 

in probability, as r — > oo. 

The above convergence is the key reason for introducing the modification 
of through the vector y* , described above Theorem 3.8. 

Next, standard moment bounds for renewal processes (see, e.g., Lemma 3.5 
of [9]) yield that \X r \ 

oo p converges to zero in probability as r — > oo. Combin- 
ing these observations, we get from (4.39) and (4.40) that (Q r ,Y r ) converge, 
uniformly over [0, p], in probability, to (q + ^-Ry*i, ^—t). In particular, this 
shows that 



(f>»(0),u r o >°) = (Q r (p),Y r (p)) 

( q + e Ry*,e y*) = (q {0) (0),v$ )) ). 



(4.45) 



Finally, we prove the convergence of (f'°,z r '°) to (^°),^°)). We will apply 
Theorem 4.1 of [29]. Note that 

(4.46) <f>°(t) = q r '°(0)+w r > (t) + Rz r > (t), te[0,9], 

where w r, ° is a "D^-valued random variable defined as w r '°(t) = (C r (t) — 
C r (p/0)l[p/r,oo)W- From (4.45) and (4.38) 

(4.47) <f'°(0) -+ ? (0) (0) and w r '° ( 



as r — > oo. 
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Next, for t G [p/r,8), write 

(4.48) z r '°(t) = H r ' (t)+L r '°(t), 

where for j G JT, 



^J.°(t)=r / (1- V, 1 ( S ))d(x*,-fJ' 1 ( S ))+r / 1^,(5) cfij' 1 ^ 



[pA,*] 



L r -°(t) = rx* I l s r,i(s)ds, 

J[p/r,t] ' 

with Sj' 1 defined in (4.17). Using calculations similar to those in the proof 
of Proposition 4.3 [see (4.22)], we get 



(4.49) sup 

p/r<t<9 



[p/r,t] > 



in probability, as r — > oo, 



for all j € J. 

Also, from (4.13) it follows that 



sup 

p/r<t<( 



[p/r,t] 



< 



Combining the above estimates, 



(4.50) 



sup (t)| in probability, as r — > oo. 

p/r<t<e 



Also, L r '°(t) = (diag(x*))C'i r '°(t), where for i G I and t G [p/r,8], 
(4.51) 



lf{t)=r 



[Pfr,t] 



Wr,t} 



1 U:> (p'-(s))<rer(s)} ds - 



Recall that z r '°(t) = for t G [0,p/r]. Hence, setting H r '°(t) = £ r '°(t) = for 
i£ [0,p/r], we have from (4.46) and (4.48) 

(4.52) q r '°(t) = q r '°{0)+w r ' (t) + RH r ' {t) + Di r ' {t), te[0,6]. 
From (4.47) and (4.50) we now have that, as r — > oo, 

(4.53) q r >°(0)+w r >° + RH r '°^q {0) +( 

in T>q. Using the definition of p r , Assumption 2.13 and elementary properties 
of renewal processes [see similar arguments in (4.22) and (4.29)], we have 
that for some C2, as r — > 00, 



sup \Q r (p r (s))-Q r (s)\>e )<C 2 

■e[p/r,0] ' 



1 



A r (A r 



Co 



,k(m+l) 



0. 
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This shows that 

(4.54) sup \q r '°(s) - q r,0 (p r (s))\ ->■ in probability, as r-> oo. 

se[o,6»] 

Using Theorem 4.1 of [29] along with (4.51), (4.52), (4.53) and (4.54), we 
now have that 

(C^ r '°,i r '°) (C,r(g (0) (o) + C),r(?W(o) + 0) as r^oo, 

as T> 1 xT>q x T>g valued random variables. Since 

r(g (0) (0)+C) = ? (0) and diag(x*)C'f(g (0) (0) + C) = z (0) , 

we get from the above display that (if' , -z r '°) =>• (<p°), .z'- ') asr-> oo. Com- 
bining this with (4.45) and observations below (4.38), we have J r [ti\ =4> J[0], 
which completes the proof of (i). 
We now prove (ii). We can write 

J r [n + 1] = (J r [n] ) (u r ' n+1 ,u r ' n+1 ,u^ n+1 ,q r ' n+1 , Z r ' n+1 )) 

J[n+l] = (J r [n] , V (n+1) , ^ n+1) , q ^ , ) ) . 

By assumption, 

(4.55) J r [n]^J[n] 

and, thus, in particular, (4.38) holds. This shows that x K ' r (n + 1) => x K ( n + 1) 
and as a consequence, using continuity properties of vj n+ \, 

„r,n+l = w x {$*r( n + l) Mn+X ) => W n+1 ( X K (n + l),U n+1 ) 

(4.56) 

In fact, this shows the joint convergence: (J r [n], p r ' n+1 ) =^> (^[n], P n+1 ). In 
particular, we have 

{v r ' n+ \Q r {{n + 1)0)) = (is r ' n +\ q r > n (9)) 

^ j q (n) = ( p(n+l) , Q(( n + 

For the remaining proof, to keep the presentation simple, we will not explic- 
itly note the joint convergence of all the processes being considered. From 
continuity of the map i? £o , we now have that 

v r ' n + 1 =# £o (Q r ((n+l)0),D r ' n+1 ) tf £0 (Q((n + l)fl-),i> +1 )) = i> +1 ). 

Next, we consider the weak convergence of v^ n+1 to v^ l+1 \ The proof is 
similar to the case n + 1 = treated in the first part of the lemma [cf. 
below (4.38)] and so only a sketch will be provided. Note that 

u r,n+l = u r,n + + ( yr ((n + l)Q + p/r) _ yr ^ + 
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Weak convergence of (q r ' n (8),v r Q ' n + z r ' n (9)) to (q {n) (9), + z^(9)) is 
a consequence of (4.55). Next, abusing notation introduced above (4.38), 
define for t £ [0, p], 

Q r (t) = r~ l Q r (r 2 9(n + 1) + rt), <f = Q r (0), 

Y r (t) = r- l {Y r {r 2 6{n + 1) + rt) - Y r {r 2 9{n + 1))). 

Processes X r , T r , T r ' , £ r are denned similarly to Y r . Then, equations (4.39) 
and (4.40) are satisfied with these new definitions. Hence, using arguments 
similar to the ones used in the proof of (4.45) (in particular, making use of 
Proposition 4.6), we have that (Y r ,Q r ) converges in distribution to 

,.(n+l) I \ 

i, Q((n + 1)9-) + -Rv( n+1 h , 

P P J 

as r — > oo. Combining the above observations, we have, as r — > oo, 
( q ^+H0),^ n+1 ) = (Q r (p),Y r (p)) 

(4.57) 

=► (Q((n + 1)9) A n) + + ^ (n+1) ) = (4 {n+1) (0)A n+1) )- 

Finally, we consider weak convergence of (cf' n+1 , z r ' n+1 ) to (q( n+1 \ z^ n+1 '). 
Similar to (4.46), we have 

f^ 1 (t) = q r ' n+1 (0) + w r ' n+1 (t) + Rz r ' n+1 (t), t € [0,9] , 

where w r,n+1 is a 2?*-valued random variable defined as 

Using (4.57) and (4.38), as r->oo, 

(4.58) ? r ' n+1 (0) + w r > n+l q( n+1 \0) + C((n + 1)9 + •) - C((n + 1)(9). 

Weak convergence of (q r ' n+1 , z r ' n+1 ) to (#( n+1 ), z^ n+l ^) now follows exactly 
as below (4.47). Combining the above weak convergence properties, we now 
have J r [n + 1] => S[n + 1] and the result follows. □ 

4.4. Proof of Proposition 4-6- We will only consider the case n = 0. The 
general case is treated similarly. Let M r = ■ From Assumption 2.13, for 
each 5 > 0, one can find C±(5) such that, for i G I, j G J, r > 1 and A; < M r , 

P(|^((fc + l)A r ) - £f (fcA r ) - oTAH > 5A r ) < 
P(|5y(r[' 1 ((fe + l)A r )) - ST(3T« 1 (fcA r )) - #t!"'*| > <5A r ) < 



(4.59) 



J j \ j W < I II "jV j V" II I / — r km ' 

— j,km ' 
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where rj' fc = Tj' 1 ((k + l)A r ) - Tj' 1 (kA r ) = (x* - e -^)A r . Denote the union, 
over all i, j, of events on the left-hand side of the three displays in (4.59), 
by HI. Then, the above estimates, along with (4.2), yield 

/ Mr \ 

(4.60) PI (J H r k \ -> as r -> oo. 

\fe=o / 
Define for k = 0, 1, . . . , M r , 

Q r ((k + l)A r ) = Q r (kA r ) + E r {(k + l)A r ) - E r ({kA r )) 

- C{S T {T r ^{{k + l)A r )) - S r {T r ' l {kA T ))) 

+ $ r (5 r (T r ' 1 ((fc + l)A r ))) - $ r (S r (T r ' 1 (ifeA r ))). 

Note that, on the set (H k ) c , we have for some C 2 > 0, 

Q r o((k + l)A r ) 

> Q r {kA r ) + a r A r - (C - P') diag(/3 r ) (x* - ^\ A r - C 2 5A r U. 

Using Assumption 2.1, we now have that for some C3, on the set (-ff£) c , for 
all r > ro, 

+ l)A r ) > Q r (kA r ) + ^Ry*A r - [c 2 5A T + C 3 ^pj U. 

Recall that Ry* > 70 li- Fix 5 small enough so that for some e\ > and 
ri > r , 

(4.61) ^70 - (c 2 5 + < y^J>£i for all r > 77. 

Then, for every k = 0, 1, . . . , M r , 

on the set (H r k ) c , Q r ((k + l)A r )>Q r ((k + l)A r ) 

4.62 

>Q r (fcA r )+eiliA r , r >n. 

Recall d\ introduced above (4.3). Let mo be large enough so that £\mo > d\. 
Then using (4.62), we get that 

M r 

(4.63) on the set f] (H r k ) c , Q r {kA r ) > diliA r , 

k=0 

for all k = itlq, . . . , M r , r > 77 . 

Next, let 

F r k = {w : inf Q r i(kA r , u) > d x A r } 

(4.64) n {u : Ql(t,u) = for some i€l,t€ [kA r , (k + l)A r ]} 
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Using estimates below (4.21), we see that 

/Mr \ 

(4.65) PjU^j^O 
Also, from (4.63), 



as r — > oo. 



\k=0 



liminfP(G^ ) >liminfP G r mo n 



(4.66) 



M r 



If 



k=0 



liminfP C](H r k Y 



1. 



\k=0 



Next, for r > r±, 
(4.67) 
Also, 



Mr 



M r 



(J <p |J +p((g; ; 



U (B£ n G£ 

\fc=rao 



n 



M r 



k=0 



(4.68) 



n 



Mr 

U (BS n cr„ 

_k=mo 

M r 

U (fllnffj 
<p( U^A+pf J (ujnGjA 



) 



fe=0 



'M r 



\k=0 



\k=mo 



where the last inequality is a consequence of the fact that on f]^ (H r k ) c , 
G k C G k+1 for k > rriQ. From (4.60) and (4.65) the above expression is seen 
to approach zero as r — > oo. Using this observation and (4.66) in (4.67), we 
now see that P(Ufc=m ~^ C as r -> oo. Finally, recalling the definition 
of B k , we have 

( Ar \ 

P(QJ(s) = 0, for some i G I and s G [m A r , rp]) = P (J 5H. 

\fc=mo / 

The proposition now follows on setting p r = m ° A . 

APPENDIX 

Lemma A.l. Lei {l^} n >i &e a sequence of random variables, with values 
in a finite set S, given on a probability space (Q, J 7 , P). Let Q be a sub-a field 
of T. Suppose that {G n }n>i is a sequence of sub-a fields of Q and {A n } n >i 
a sequence of {Qn} -adapted, M. d -valued random variables such that 

nY n = C\Gn)=Pn,d X n), n > 1, C G S, 



52 



A. BUDHIRAJA AND A. P. GHOSH 



where p n ^ : R d -> [0, 1] are measurable maps such that S^ g gp n ,f (x) = 1 for all 
x G n > 1. T/ien there is a sequence of S-valued random variables {Y n } 
defined on an augmentation of (Cl, J 7 , P) such that 



\Y n = as v y^ 1 ) =Pn, c (X n ), n > i,c € s, 



Proof. By suitably augmenting the space, we can assume that the 
probability space (Cl, T, P) supports an i.i.d. sequence {U n } n >i of Uniform 
[0, 1] random variables, independent of Q. Let, for n > 1, £ G S, a n ^, 6 n ^ : W 1 — > 
[0, 1] be measurable maps, such that for all x G ]R rf : 

(0 - a n,c( X ) =Pn,((x), Tl > 1, ( G §. 

(ii) [a n>c (x),6 niC (x))n[a n)C i(x) J 6 n)C i(x)) = J CC'eS, CtK', ™> L 
(hi) Uce§[ an >c( x )' ^n.cO^)) = 1)- The resu lt follows on defining 



□ 



A.l. Proof of Lemma 3.7. From (2.38) we have that for some C\ € 
(0,oo), 

(A.i) IQ'-Q'locT^Cii^-y 2 !^. 

Thus, for some C2 G (0, 00), 



(A.2) 



E 



f e~"' t h-Q 1 (t)dt-t [ e~" lt h-Q 2 (t)dt 
Jo Jo 



< CoEly 1 -Y 2 



oo,T- 



Next, for t>0 and £ = 1,2, 

(A.3) Q\t + T)= T(Q\T) + ((T + •) - ((T)). 

Using Assumption 2.15 and (A.l), we now have that for all t > 0, 

(A.4) IQ 1 ^ + T) - Q 2 (t + T)\< L\Q l (T) - Q 2 (T)\ < LC\\Y l - Y 2 \^ T . 

This shows that, for some C3 G (0,oo), 



(A.5) 



E 



poo roc 

\ e-^ t h-Q 1 (t)dt-t / e-^h-Q 2 (t)dt 
Jt Jt 

e~^p-dU 2 (t) 



<Czt\Y l -y 2 |oo,T. 



Next, for some C4 G (0, 00), 



[0,T] 



< 



[0,T] 



(A.6) 



1^(0) - U 2 (0)\ + e-"> T \U l (T) - U 2 (T)\ 



+ 7 / \U\t)-U 2 (t)\dt 



<c 4 |y' 1 -y' 2 |oo,T. 
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Next, note that for S >T, z = 1,2, 

(T,S] 

(A.7) 

= 7 / e~^p ■ [U\t) - U l {T)) dt + e-^p ■ [U^S) - U' L {T)]. 

Also, using Assumption 2.15 and (2.39), for some C5 G (0,oo), 

\U\t) - U\T)\ < C 5 \((T + ■)- C(T)\oo,t-T, t > T, 

which shows that, for i = 1, 2, 

E|e^V [tP(S) -U\T)}\ ->-0 asS^oo. 

Combining this observation with (A.7), we now have, on sending S — > 00, 
that for z = 1,2, 



E / e-^p ■ dtP(t) = 7E / e-^p ■ [Wit) - U\T)\ dt. 

J{T,oa) JT 

Thus, for some Cq G (0, 00), 

e-^p ■ dt) 1 (t) - [ e-^p ■ dU 2 (t) 

(T,oo) J(T,oo) 



<7bl / e-^i^ctj-tr^TOl-I^W-^r)]!* 

J(T,oo) 

<C 6 E ( e-^\Q\T)-Q 2 (T)\dt, 

J{T,ao) 

where the last equality follows on using Assumption 2.15 and (A. 3). Com- 
bining this with (A.l), we now have that 



E 



e-^p-dU\t)- / e-^p-dU 2 (t) 

(T,oo) J(T,oo) 



The result now follows on combining the above estimate with (A. 2), (A. 5) 
and (A.6). 

A. 2. Proof of Proposition 4.1. It is immediate from the construction 
that T r satisfies (i)-(iii) of Definition 2.7. We now verify that, with Q = 
a{Ui,i > 1}, 

(A.8) T r satisfies (iv). 

The proof of (A.8) is similar to that of Theorem 5.4 in [9], which shows that 
if a policy satisfies certain natural conditions (see Assumptions 5.1, 5.2, 5.3 
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therein), then it is admissible (in the sense of Definition 2.7 of the current 
paper). The policy T r constructed in Section 4.1 does not exactly satisfy 
conditions in Section 5 of [9], but it has similar properties. Since most of the 
arguments are similar to [9], we only provide a sketch, emphasizing only the 
changes that are needed. For the convenience of the reader, we use similar 
notation as in [9]. Also, we suppress the superscript r from the notation. 
Recall from (4.8) that 

(A.9) Tj(t) = J l {Q ^ u){u)>0} l {Q ^ U){p{u))>e{u)} fj 1] (u)du, 

je3,t>0. 

In particular, Assumption 5.1 of [9] is satisfied. In view of (2.1), the integrand 
above has countably many points (a.s.) where the value of Tj changes 
from to 1 (or vice versa). Denote these points by {Y^g^. Set Yq = 0. We 
refer to these points as the "break-points" of T. Break-points are boundaries 
of the intervals of the form {[nr 2 9, (n + l)r 2 6)} n or those of subintervals of 
length Vj' or V 2 ' ' n for some k,i,n [see (4.11), see also (4.4) for V i ' 
or V-H that are used to define the policy T. Next, define {Y°}^ 6 n as the 
countable set of (random) "event-points" as defined in [9] (denoted there as 
{T^}£ g pj ). These are the points where either an arrival of a job or service 
completion of a job takes place anywhere in the network. Combining the 
event-points and the break-points, we get the set of "change-points" of the 
policy T denoted by {T^}: 

{T,} = {T°}U{T]}. 

We will assume that the sequence {T^} [resp. {Y°}, {Tj}] is indexed such 
that [resp. T9, Y]] is a strictly increasing sequence in I. 

As noted earlier, [9] uses the notation {Y^}, instead of {Y^}, for event 
points. We have made this change of notation since {Y^} here plays an 
identical role as that of event-points in the proof of [9]. In particular, it is 
easily seen that Assumption 5.2 of [9] holds with this new definition of {Y^}. 
We will next verify Assumption 5.3 (a nonanticipativity condition) of [9] in 
Lemma A. 2 below. 

For i e I and £ G N , let u[ = ^(^(T^ + 1) - T e . Thus, u\ is the residual 
(exogenous) arrival time at the ith buffer at time T^, unless an arrival of the 
ith class occurred at time Tg, in which case it equals u\ = UiiEi^Tg) + 1). 
Similarly, for j € J, t £ No, define = r^(«Sj(T^) + 1) — T^. Next, write 

T(t) = J *T(s)ds, where T is right continuous. For i 6 I, set Qifi = 0, and 
for t > 1, Q i4 = Qi(? e ). Also, for j £ J, and £ > 0, let ff = fj(T e ). Let 
ff 1 = 0. Finally, define for t > 0, 

(A. 10) x £ = {C^Vj < , vf , Qf , if' 1 : i G I, j £ I, {Ui : i G N}) : l' = 0, . . . , 1} . 
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The definition of x e above is similar to that in [9] , with the exception of the 
sequence {Ui :i£N}. This enlargement of the collection x is needed due to 
the randomization step, involving the sequence {U{}, in the construction of 
the policy [see (4.10)]. In [9], part (iv) of the admissibility requirement (for 
the smaller class of policies considered there) was in fact shown with respect 
to a smaller filtration, namely, F 1 ' ((m,n)). Here, using the above enlarge- 
ment, we will show that part (iv) holds (for the policy in Section 4.1) with 
J^{(jn,n)) = F 1 '((m, n)) V {Ui}. In Lemma A. 2 below, we prove that T(T^) 
is a measurable function of x , for all £ G No- This shows that T satisfies 
Assumptions 5.1-5.3 of [9] with the modified definition of and x ■ Now 
part (iv) of the admissibility requirement [i.e., (A. 8)] follows exactly as the 
proof of Theorem 5.4 of [9]. This completes the proof of the proposition. 

Lemma A. 2. T(Y^) is a measurable function of x , f or a ^ £ £ ^o- 

Proof. Let for m G No, L m = (T^ +1 - T^) denote the length of the 
mth break-point interval. Define k° = and = max{m > : < T^} for 
£ G No- Hence, k denotes the number of break-points that preceded the 
£th change-point, and T \ is the "last" break-point before the £t\i change- 
point T e (note that k 1 < £) for all fe% Also, define for £ G N , A e = 
(T^ +L K i — Ti) as the "residual" time for the next break-point after T^. In 
particular, A^ = implies that itself is a break-point. By definition of T 
[see (4.10)], it follows that T 1 ^, L K e, and, hence, A £ are all measurable 
functions of x^ f° r £ No- Summarizing this, we get 

(A. 11) k 1 , A , Y* £ are measurable functions of x f° r ^ £ f^o- 

Using notation from [9], let Ji = {j G J : o"i(j) = i} be the set of all activities 
that are associated with the buffer i and, for a G {0, 1} J , Ji(a) be as defined 
by equation (5.2) of [9]. Then Ji(T{t)) denotes all activities in Ji that are 
active at time t > 0, under T. Clearly, for £ G No, 

(A.12) T e = T/_i + minmin^-Vf- 1 , v^ 1 :j G J^ 1 )}. 

For i £ I, let Zf be the indicator function of the event that at the change- 
point Ti an arrival or service completion occurs at buffer i. More precisely, 
for i G I and £ > 0, 



(A.13) 



\ 1, if mmiu^^-.jeJiif*- 1 )} 

= minmin{A^ 1 ,4- 1 ,^- 1 :j G JH^" 1 )}, 

i'ei J 

, 0, otherwise. 



From (A.ll) and (A.10), it follows that 

(A. 14) both Zf , Ti are measurable functions of x , £ G Nq. 
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Using (4.10) and the construction below it, along with (A. 14), it is easily 
checked that f^(T e ) is a measurable function of x ■ Next, for j G JJ, 

(A.15) Tj(T e ) = l{Q^ U) (T e )>o}^{Q^ lU) ( P (T e ))>e(r e )}T^\T e ). 

From (A. 14) and (A. 10), 0(T^) and Q ai ^^(Ti) are x' measurable, thus, so is 
the first indicator in the above display. Also, since p(T^) is either or T 1 ^— 
depending on whether is in |J n< X^in) or not — and both Q ai ^(Ti) 

and Q ai (j}(T^ e ) are x measurable, we see that the second indicator in (A.15) 
is x^ measurable as well. The lemma follows on combining the above obser- 
vations. □ 

A. 3. Proof of Lemma 4.2. Since h n is equicontinuous, pre-compactness 
of (/ n ,5 f?1 ,/i ri ,7 n ) is immediate. Suppose now that (f n ,g n ,h n ,j n ) converges 
(in £> 3I+1 ), along some subsequence, to (f,g,h,j). Then 7(t) = t for t > 
and f,g,h^C l . Also, for suitable measurable maps hi : [0, oo) — > [0, 1] , i G I, 

hi(t)= / hi(s)ds, iel,t>0. 
J[o,t] 

If ip : [0, oo) — > M. is a continuous map with compact support, then, along the 
above subsequence, 

/ i J (fi l ( s )) 1 {f n (y(s))<e n } d s^ i/j(fi(s))hi(s)ds, t>0,iel. 

J[0,t] J[0,t] 

Suppose now that supp(V') C (5, oo) for some 5 > 0. Then the left-hand side 
of the above display converges to and so for all t, i, Jj Q ^ ijj(fi(s))hi(s) ds = 
for such ip. Since 6 > is arbitrary, we get 

/ hf i ( 3 )=o}hi(s)ds = h i (t). 
J[0,t] 

The result follows. 

Acknowledgment. We thank an anonymous referee for pointing us to the 
paper [21]. 
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